Louwrentius

Tunneling Elixir cluster network traffic over Wireguard

2023-11-07T12:00:00+01:00

Introduction

The other day I was supporting a customer with an Elixir-based platform that would make use of Elixir libcluster, so messages on one host can be passed to other hosts. This can - for example - enable live updates for all users, even if they are not communicating with the same application server.

Encryption

Elixir's libcluster does support encrypted communication using TLS certificates however I was struggling with the help of an application developer to make it work.

"severity":"warn","message":"[libcluster:example] unable to connect to :\"app@Host-B\"

I'm absolutely open to the idea that we did something wrong and certificate-based encryption will work, but we were time-constrained and we decided to opt for another solution that seemed simpler and easier to maintain.

Wireguard as the encrypted transport

I deployed a Wireguard mesh network between all application servers using Ansible, which was straight forward. We just provisioned all hosts into the /etc/hosts file to keep things simple.

In the table below, we show a simplified example of the setup.

Hostname	IP-address	Wireguard Hostname	Wireguard IP-address
Host-A	10.0.10.123	Host-A-wg	192.168.0.1
Host-B	10.0.11.231	Host-B-wg	192.168.0.2

The Elixir applications would only know about the Host-A|B-wg hostnames and thus communicate over the encrypted VPN tunnel.

The problem with wireguard and libcluster

The key issue with libcluster is that when Host-A connects to Host-B, it uses the DNS hostname Host-B-wg hostname. But the actual hostname of Host-B is - you guess it: 'Host-B'. This means there is a mismatch and for reasons unknown to me, the libcluster connection will fail.

So the target hostname as configured in libcluster must match the hostname of the actual host! Since libcluster seems to make usage of domain names mandatatory, using IP-addresses was not an option.

If we would point Host-B to it's Wireguard IP-address (192.168.0.2), the problem would be solved. However, in that case, Wireguard doesn't know about the external 10.0.11.231 IP address and also tries to connect to the non-existing 192.168.0.2 address. So the Wireguard tunnel would never be created.

The solution

The solution is not that elegant, but it works. We still point the Host-B domain name to the wireguard IP address of 192.168.0.2 but we create an additional DNS record specifically for Wireguard, pointing to 10.0.1.231, so it can setup the VPN tunnel.

This is what /etc/hosts looks like on Host-A:

10.0.10.123 Host-A
192.168.0.2 Host-B
10.0.11.231 Host-B-wg

And this is what /etc/hosts looks like on Host-B:

10.0.11.231 Host-B
192.168.0.1 Host-A
10.0.10.123 Host-A-wg

Evaluation

Although all choices are a tradeoff, for us, the Wireguard-based solution makes most sense. Especially now that we have an encrypted tunnel between all hosts and any future communication between hosts can thus be encrypted without any additional effort.

IKEA $50 VINDSTYRKA vs. $290 Dylos air quality monitor

2023-09-17T12:00:00+02:00

This is a brief article in which I compare the IKEA VINDSTYRKA $50 air quality monitor (PM2.5) with a $290 air quality monitor made by Dylos to see if it's any good.

Context

If you care about indoor air quality, you may already own a CO2 to determine if it's time to ventilate your space a bit¹.

But a CO2 monitor doesn't tell you anything about the amount and size of particulate matter in the air.

Of particular interest are very fine particles, in the "PM2.5" category. Those particles are 2.5 micrometers or smaller in diameter can embed themselves deep inside the lungs and cause health issues.

Both air quality monitoring devices are specifically measuring PM2.5 particulate matter, so that's what we will focus on in this test.

DYLOS DC1100 PRO

I bought a Dylos DC1100 Pro in 2014 as I was quite interesting in the topic of air quality at that time. As I had to import the device, I believe I had to pay around 400 Euros for it but it's now for sale in the US for around $290.

click on the image for a picture of the back

I specifically chose this model because it has a serial port, which allows me to log data and maybe spot some trends. I was thinking about using this data to control my air circulation system in my home, but I never got around to building this.

This device (without serial port) is also explored in-depth by Matthias Wandel, who many of you probably know from his 1.7M subscribers Youtube channel. Tip: he shows the inside of the device.

Note that this video is from 10 years ago and I find it remarkable that the Dylos 1100 Pro is still sold - seemingly unmodified - over the years.

The IKEA VINDSTYRKA

Recently, I discovered that IKEA is now selling the VINDSTYRKA air quality monitor with support for Zigbee. The product is intended to be used with IKEAs range of air purifiers, to better finetune the behaviour of those devices.

The device measures PM2.5 particulate matter and also monitors temperature and humidity. All data is exposed over Zigbee. I've not tested this myself but I wonder how long it would last on a battery bank as it's USB powered.

Due to the low price tag, I decided to compare this $50 device (€40) with my Dylos. I think it's quite an interesting device because the Zigbee support allows you to integrate the device in home automation and log data, if you have a need for that.

Data logging setup

De Dylos device is a bit of a pain, because the measurement values are in particulates per square foot, so I had to find a proper conversion formula, which I found in this paper (page 17). The formula is:

PM2.5 Dylos DC1100 (μg/m3) = (particles > 0.5 μm minus particles > 2.5 μm)/250.

A Raspberry Pi 3B+ is running a Python script that reads the data from the serial port, converts it to PM2.5 values using the previous mentioned formula and transmits it into an InfluxDB + Grafana server.

To log the VINDSTYRKA data, I used a Sonoff Zigbee receiver on a Raspberry Pi 4b+. I installed zigbee2mqtt as a docker container, Mosquitto MQTT server and Telegraf+MQTT-client to submit the data into InfluxDB, which sounds more convoluted than it actually was.

Test method

I just let both devices run for a few days in close proximity to each other in my living room. I kept a balcony door open 24/7. I also created a bit of smoke at some point just to observe how the devices would respond and how much they would deviate from each other. Nothing too scientific, to be frank.

Test result

I've plotted the data from the Dylos and the Ikea device in the same graph and I think the results are quite straightforward. The peak in the middle was my 'smoke test'.

Original click on the image for a larger version

Updated click on the image for a larger version

After a few days I noticed a clear deviation between the VINDSTYRKA and the Dylos DC1100 Pro at certain time intervals. I have no real explanation for this deviation and I can't tell which device shows 'correct' data.

If I try to follow the AQI PM2.5 values for my city, the VINDSTYRKA seems to under report and the Dylos seems to over-report PM2.5 particulate matter.

Evaluation

Based on my test, I think the VINDSTYRKA is good enough, looking at how closely the measurements track the results of the Dylos.

Update September 19th 2023 Based on the new graph data, it seems the Dylos and VINDSTYRKA are less in agreement over absolute PM2.5 values. I'm not sure what to make of it.

As both devices still seem to agree on basic trend data, I would say that they still operate in the same ballpark.

Maybe you are a home automation enthusiast and you've managed to automate this process. ↩

My solar-powered blog is now on Lithium Iron Phosphate

2023-05-19T12:00:00+02:00

In my last blog post I discussed how a small solar project - to power this blog on a Raspberry Pi - escalated into a full-blown off-grid solar setup, large enough to power the computer I use at the moment to write this update¹. In this update, I want to discuss my battery upgrade.

For me, the huge lead acid battery (as pictured below) was always a relatively cheap temporary solution.

A 12 Volt 230 Ah lead-acid battery

Lead-acid batteries are not ideal for solar setups for multiple reasons, but the most problematic issue is the slow charging speed. I only have a few hours of direct sunlight per day due to my particular situation and the battery just could not absorb sunlight fast enough.

For the last 5-7 years, the go-to battery chemistry for solar is LiFePO4 or lithium iron phosphate as a replacement for lead-acid batteries. This battery chemistry is not as energy-dense as Lithium-ion, but the upside is price and safety. In particular, LiFePO4 cells aren't as volatile as Lithium-ion cells. They may start outgassing, but they don't start a fire.

More importantly for my situation: LiFePO4 batteries can charge and discharge at much higer rates than lead-acid batteries². It's possible to charge LiFePO4 cells with a C-rate of 1! This means that if a battery is rated for 100Ah (Ampere-hours) you can charge with a current of 100 Ampere! My solar setup will never come even close to that number, but at least it's good to have some headroom.

A single 3.2 volt 230Ah Lithium Iron Phosphate prismatic cell

I did contemplate buying an off-the-shelf battery but I decided against it. You have no control over the brand and quality of the LiFePO4 cells they use and more importantly, what's the fun in that anyway?

So I decided to order my own cells and build my own 12 Volt LiFePO4 battery consisting of four cells in series (4S) as my existing system is also based on 12 Volt. Other common configurations are 8S (24 Volt) and 16S (48 Volt³).

It turned out that I could just buy my cells locally in The Netherlands (instead of China) because of a company that specializes in batteries (no affiliate). As the price was right, I bought effectively 3 KWh for just shy of 500 Euros.

I decided to buy B-grade cells as those are cheaper than A(utomotive)-grade cells. I might have gone for A-grade cells as not to risk anything if I would build a more serious battery bank for my whole home. Yet a lot of people report no significant differences between A-grade and B-grade LiFePO4 cells for solar battery banks so in the end, it's all about your particular apetite for risk.

Just buying cells and putting them in series (in my case 4S) is not enough, a BMS or battery management system is needed, which you put in series with the battery on the negative terminal. I ordered a 100A Daly BMS from China which works fine. I'm even able to use Python to talk with the Daly BMS over bluetooth to extract data (voltages, current, State of Charge and so on).

The BMS is critical because it protects the cells against deep discharge and overcharging. In addition, the BMS tries to keep the voltage of the cells as equal as possible, which is called 'balancing'. Charging stops entirely when just one of the cells reach their maximum voltage. If other cells have a much lower voltage, it means that they can still be charged but the one cell with the high voltage is blocking them from doing so. That's why cell balancing is critical if you want to use as much of the capacity as possible.

The Daly BMS is quite bad at cell balancing so I've ordered a separate cell balancer for $18 to improve cell balancing (yet to be installed).

Ikea sells a Kuggis 32cmx32cmx32cm storage box that seems to be perfect for my small battery. As it has two holes on the sides, I just routed the positive and negative cables through them.

Now that I've put this battery in place I've seen a huge improvement regarding solar charge performance.

I've actually potentially created a new problem: my solar charge controller can only handle about 400 Watts of solar power at 12V and my setup is quite close to reaching this output. I may have undersized my solar charge controller and it has come back to bite me. For now, I'm going to just observe: if that peak of 400 Watts is only reached for a brief time - as it is right now - I don't think I'm going to upgrade my solar charge controller as that would not be worth it.

As we are still in May, my best yield is 1.2 KWh per day. Although that's paltry as compared to regular residential solar setups, that 1.2 KWh is more than a third of my battery capacity and can run my computer setup for 10 hours, so for me it's good enough.

It's funny to me that all of this started out with just a 60 Watt solar panel, a 20 Euro solar charge controller (non MPPT) and a few 12V 7Ah lead acid gel batteries in parallel.

I think it's beyond amazing that you can now build a 15KWh battery bank complete with BMS for less than €3000. For that amount of money, you can't come even close to this kind of capacity.

For context, it's also good to know that the longevity of LiFePO4 cells is amazing. A-grade cells are rated for 6000 cycles ( 16+ years at one cycle per day ) and my vendor rated B-grade cells at 4000 cycles (~11 years).

Maybe my battery build may inspire you to explore building your own battery. LiFePO4 cells come in a whole range of capacities, I've seen small 22Ah cells or huge 304Ah cells so you can select something that fits your need and budget.

If you're looking for more information: there are quite a few Youtubers that specialise in building large battery banks (48 Volt, 300Ah, ~15KWh) to power their homes and garages.

Although Will Prowse reviewed LiFePO4 cells in the past, he currently focusses mostly on off-the-shelf products, like "rack-mount" batteries and inverter/chargers.

I also like the off-grid-garage channel a lot, the channel as tested and explored quite a few products.

Harrold Halewijn (Dutch) also has quite a few videos about solar setups in general and solar battery setups. He's really into automation, in combination with flexible (next-day) energy prices.

Also in Dutch, a cool article about some people building their own large-scale home storage batteries (15KWh+)

Another Dutch person build a solar power factory with a battery capacity of 128 KWh for professional energy production. Truely amazing.

The Hacker News thread about this article.

https://news.ycombinator.com/item?id=35596959#35597492 ↩
To fully charge a lead-acid battery, the charging process spends a lot of time in the contstant-voltage phase, the voltage is kept constant so as the battery charges further, the charging current goes down, so the charge process slows down. More info can be found here ↩
It seems to me that most batteries build for home energy storage systems are standardising on 48 volt. ↩

I made my blog solar-powered, then things escalated

2023-04-17T12:00:00+02:00

In 2020 I wondered if I could run my blog on solar power, being inspired by Low-tech Magazine, doing the same thing (but better)¹. The answer was 'yes', but only through spring and summer.

I live in an apartment complex in The Netherlands and my balcony is facing west. This means it only receives direct sunlight from 16:00 onward during spring and summer. Most of the time, the panels only get indirect sunlight and therefore generate just a tiny fraction of their rated performance. The key issue is not solar, but the west-facing balcony (it should ideally be facing south).

original solar panel

It's fair to say that my experiment isn't rational because of the sub-optimal solar conditions. Yet, I'm unreasonably obsessed by solar power and I wanted to make it work, even if it didn't make sense from an economic or environmental perspective².

When I wrote my blog about my solar-powered setup, I was already on my second iteration: I started out with just a 60 Watt panel and a cheap $20 solar controller³. That didn't even come close to being sufficient, so I upgraded the solar controller and bought a second panel rated for 150 Watt, which is pictured above. With the 60 Watt and 150 Watt panels in parallel, it was still not enough to keep the batteries charged in the fall and winter, due to the west-facing balcony.

A Raspberry Pi 4B+ consumes around ~3.5 Watt of power continuously. Although that sounds like a very light load, if you run it for 24 hours, it's equivalent to using 84 Watts continuously for one hour. That's like running two 40 Watt fans for one hour, it's not insignificant and it doesn't even account for battery charging losses.

So 210 Watt of solar (receiving mostly indirect sunlight) still could not power my Raspberry Pi through the winter under my circumstances. Yet, in the summer, I had plenty of power available and had no problems charging my iPad and other devices.

As my solar setup could not keep the batteries charged from October onward, I decided to do something radical. I bought a 370 Watt⁴ solar panel (1690 x 1029 mm) and build a frame made of aluminium tubing⁵. Solar panels have become so cheap that the aluminium frame is more expensive than the panel.

Even this 370 Watt panel was not enough during the gloomy, cloudy winter days. So I bought a second panel and build a second tube frame. Only with a 740 Watt rated solar panel setup was I able to power my Raspberry Pi through the winter⁶.

I didn't create this over-powered setup just to power the Raspberry Pi during the winter. I knew that solar performed much better during spring and summer and I wanted to capture as much of that energy as possible. The real goal was to go beyond powering the Pi and power my computer desk, which includes an Intel Mac Mini, two 1440p 27" displays and some other components (using around 100 Watt on average)⁷.

I would not be able to power my desk 24/7 but I would be happy if I can work on solar power for a few hours every other day during spring and summer. I also wanted to light my house in the evening using this setup.

The original solar setup was enough to power the Raspberry Pi and charge an iPad in the spring/summer. The solar charge controller could not handle the increased solar capacity and needed replacement. So I decided to build a new setup inspired by Will Prowse solar demo setups, which is pictured below:

the latest iteration of my solar setup

First a brief disclaimer: I'm a hobbyist, not an expert (if you didn't notice already). I have no background in electrical systems. I've tried to make my setup safe, but I may have done things that are not recommended.

My setup is a 12-volt system⁹. The drawback of a 12-volt system is the relatively large currents required to charge the battery and power the inverter. This requires thicker, more expensive cabling to prevent energy losses in the cabling⁸.

Most components are self-explanatory, except for the shunt. This device precisely measures battery voltage and how much current is going in and out of the battery. The solar charge controller and the shunt are linked together in a bluetooth network, so the solar controller uses the precise voltage and current information from the shunt to regulate the battery charging process.

The solar controller, inverter and shunt have VE.direct interfaces (Victron-specific) which I use to collect data. I'm using a Python VE.direct module to gather this data, which just works without any issues. My Python script dumps the data into InfluxDB and I use Grafana for graphs (see below). The script also updates the 'solar status' bar to the right (or bottom for mobile users).

this image is updated periodically

The LCD display is just for fun, and mostly to keep an eye on the battery charge state.

the 20x4 LCD screen

The LCD screen is managed by the same python script that dumps the VE.direct data into Grafana. I focus on two metrics in particular. First of all the daily solar yield as a percentage: 100% means the load has been compensated by solar and anything higher means an energy 'profit'. In the bottom right we see the charger status (Bulk): if it's on 'Float' the battery is full. I tend to wait for the battery to recharge to a 'float' status before I use the inverter again.

Let's talk about the battery. I've chosen to use a large used lead-acid battery even though Lithium (LiFePO4) batteries beat lead-acid in every metric.

Update May 2023: I have since upgraded to Lithium Iron Phosphate, see this blogpost for more information.

A 12 Volt 230 Ah lead-acid battery¹⁰

I bought the battery¹¹ second-hand for €100 so that's not a significant investment for a battery. Although it's a bit worn-down and the capacity is reduced, it is still good enough for me to run my computer setup for 10 hours after a full charge¹². In practice, I won't use the battery for more than four to five hours at a time because recharging can take multiple days and lead-acid batteries should ideally be fully recharged within 24 hours or their aging is accelerated.

The lead-acid battery also serves another purpose: is a relatively cheap option for me to validate my setup. If it works as intended, I might opt to upgrade to lithium (LiFePO4) at some point.

Until recently, switching between the grid and solar for my computer setup was quite cumbersome. I had to power down all equipment, connect to the inverter and power everything up again. That got old very quickly. Fortunately, I stumbled on an advertisement for a Victron Filax 2 and it turns out that it does exactly what I need.

The Filax 2 switches between two 230 Volt input sources without any interruption, like a UPS (Uninterruptible power supply). Now that I've installed this device, I can switch between solar and grid power without any interruption. Brand new, The Filax 2 costs €350 which was beyond what I wanted to spend, but the second-hand price was acceptable.

My solar setup is not something that I can just turn on and forget about. I have to keep an eye on the battery, especially because lead-acid should ideally be recharged within 24 hours.

Happy case: the battery is full and my computer desk is 100% solar-powered

It's now April 2023 and my setup seems promising. Peak output of the two 370 Watt solar panels facing west was 230 Watts. Only for a very short period, but it makes me confident for spring and summer. I could automate enabling and disabling the inverter, with a relay and some logic in Python, but for now I'm good with manually operating the inverter.

You may have noticed that I've used a lot of Victron equipment¹³. Mostly because it seems high-quality and the data interfaces are documented and easy-to-use. The inverter was also chosen because of the low parasitic load (self-consumption) of around 6 watt. Victron equipment is not cheap. Buying Victron gear second-hand can save a lot of money.

Speaking of cost, if I include all the cost I made, including previous solar projects and mistakes, I think I spend around €2000.

That's all I have to say about my hobby solar project for now.

Link to Hacker News thread about this article.

Their attempt was quite serious and precise. They accounted for the energy used to produce the equipment. They went as far as dithering images to reduce bandwidth and thus energy usage. ↩
The cost can never be reclaimed by the electricity savings. Also, the energy produced to make all the components would never be recovered due to my west-facing setup. ↩
never buy those cheap non-MPPT solar charge controllers unless you really know what you are doing. You are better off with a MPPT controller which is much better at getting the most energy out of a solar panel. ↩
Jinko half cut 120 cell 370 WP JKM370N-6TL3-B ↩
I have absolutely no experience with designing and building aluminium tube frames. After you're done laughing at this contraption, if you have a better, more efficient design, I'm still interested. ↩
I may have cheated once by recharging the battery from the grid just to protect it against accelerated aging due to being in a prolonged (partially) discharged state. ↩
Suddenly you realise that making the background on all monitors black saves ~20 Watt. My blog should be dark-themed to reduce energy usage 😅 ↩
I've actually oversized the battery cabling for safety reasons. If a length of cable isn't rated for the current flowing through it, it becomes a resistor, generating heat, which can cause a fire so I want to be carefull. ↩
If you ever intend to build some kind of solar setup yourself, consider a 24 Volt or ideally an 48 Volt system to reduce currents and thus save on cabling cost. ↩
The + and - pole are temporary uncovered for this picture, but normally they are covered to prevent a short-circuit if anything would fall on the poles. ↩
A sealed lead-acid battery like the one I'm using is safe and won't release any (explosive) gasses unless overcharged or abused. ↩
The inverter uses a dynamic load algorithm to prevent deep discharge of the battery. Ideally a lead-acid battery should never be discharged beyond 50% of capacity and it seems to work perfectly. Dumb inverters, just discharge until 10.5 volt under load, which means the battery is almost depleted, causing rapid aging and significantly reduced life-span. ↩
No, I'm not sponsored by Victron, I wish 😅💸 ↩

Benchmarking cheap SSDs for fun, no profit (be warned)

2023-03-26T12:00:00+02:00

The price of Solid-state drives (SSDs) has dropped significantly over the last few years. It's now possible to buy a 1TB solid-state drive for less than €60. However, at such low price points, there is a catch.

Although cheap SSDs do perform fine regarding reads, sustained write performance can be really atrocious. To demonstrate this concept, I bought a bunch of the cheapest SATA SSDs I could find - as listed below - and benchmarked them with Fio.

Model	Capacity	Price
ADATA Ultimate SU650	240 GB	€ 15,99
PNY CS900	120 GB	€ 14,56
Kingston A400	120 GB	€ 20,85
Verbatim Vi550 S3	128 GB	€ 14,99

I didn't have the budget to buy a bunch of 1TB of 2TB SSD, so these ultra-cheap, low capacity SSDs are a bit of a stand-in. I've also added a Crucial MX500 1TB (CT1000MX500SSD1) SATA¹ SSD - which I already owned - to the benchmarks to see how well those small-capacity SSDs stack up to a cheap SSD with a much larger capacity.

Understanding SSD write performance

To understand the benchmark results a bit better, we discuss some SSD concepts in this section. Feel free to skip to the actual benchmarks if you're already familiar with them.

SLC Cache

SSDs originally used single-level cell (SLC) flash memory, which can hold a single bit and is the fastest and most reliable flash memory available. Unfortunately, it's also the most expensive. To reduce cost, multi-level cell (MLC) flash was invented, which can hold two bits instead of one, at the cost of speed and longevity². This is even more so for triple-level cell (TLC) and quad-level cell (QLC) flash memory. All 'cheap' SSDs I benchmark use 3D v-nand³ TLC flash memory.

One technique to temporarily boost SSD performance is to use a (small) portion of (in our case) TLC flash memory as if it was SLC memory. This SLC memory then acts as a fast write cache⁴. When the SSD is idle, data is moved from the SLC cache to the TLC flash memory in the background. However, this process is limited by the speed of the 'slower' TLC flash memory and can take a while to complete.

While this trick with SLC memory works well for brief, intermittent write loads, sustained write loads will fill up the SLC cache and cause a significant drop in performance as the SSD is forced to write data into slower TLC memory.

DRAM cache

As flash memory has a limited lifespan and can only take a limited number of writes, a wear-leveling mechanism is used to distribute writes over all cells evenly, regardless of where data is written logically. Keeping track of this mapping between logical and physical 'locations' can be sped up with a DRAM cache (chip) as DRAM tend to be faster than flash memory. In addition, the DRAM can also be used to cache writes, improving performance. Cheap SSDs don't use DRAM cache chips to reduce cost, thus they have to update their data mapping tables in flash memory, which is slower. This can also impact (sustained) write performance. To be frank, I'm not sure how much a lack of DRAM impacts our benchmarks.

Benchmark method

Before I started benchmarking I submitted a trim command to clear each drive. Next, I performed a sequential write benchmark of the entire SSD with a block size of 1 megabyte and a queue depth of 32. The benchmark is performed on the 'raw' device, no filesystem is used. I used Fio for these benchmarks.

Benchmark results

The chart below shows write bandwith over time for all tested SSDs. Each drive has been benchmarked in full, but the data is truncated to the first 400 seconds for readability (performance didn't change). The raw Fio benchmark data can be found here (.tgz).

click for a larger image

It's funny to me that some cheap SSDs initially perform way better than the more expensive Crucial 1TB SSD⁵. As soon as their SLC cache runs out, the Crucial 1TB has the last laugh as it shows best sustained throughput, beating all cheaper drives, but the Kingston A400 comes close.

Of all the cheap SSDs only the Kingston shows the best sustained write speed at around 100 MB/s and there are no intermittent drops in performance. The ADATA, PNY and Verbatim SSDs show flakey behaviour and basically terrible sustained write performance. But make no mistake, I would not call the performance of the Kingston SSD, nor the Crucial SSD - added as a reference - 'good' by any definition of that word. Even the Kingston can't saturate gigabit Ethernet.

The bandwidth alone doesn't tell the whole story. The latency or responsiveness of the SSDs is also significantly impacted:

click for a larger image

The Crucial 1TB SSD shows best latency overall, followed by the Kingston SSD. The rest of the cheap SSDs show quite high latency spikes and very high latency overall, even when some of the spikes settle, like for the ADATA SSD. When latency is measured in seconds, things are bad.

To put things a bit in perspective, let's compare these results to a Toshiba 8 TB 7200 RPM hard drive I had lying around.

click for a larger image

The hard drive shows better write throughput and latency⁶ as compared to most of the tested SSDs. Yes, except for the initial few minutes where the cheap SSDs tend to be faster (except for the Kingston & Crucial SSDs) but how much does that matter?

As we've shown the performance of a hard drive to contrast the terrible write performance of the cheap SSDs, it's time to also compare them to a more expensive, higher-tier SSD.

click for a larger image

I've bought this Samsung SSD in 2019 for €137 euro, so that's quite a different price point. I think the graph speaks for itself, especially if you consider that this graph is not truncated, this is the full drive write.

Evaluation & conclusion

One of the funnier conclusions to draw is that it's beter to use a hard drive than to use cheap SSDs if you need to ingest a lot of data. Even the Crucial 1TB SSD could not keep up with the HDD.

A more interesting conclusion is that the 1TB SSD didn't perform that much better than the small cheaper SSDs. Or to put it differently: although the performance of the small, cheap SSDs is not representative of the larger SSD, it is still quite in the same ball park. I don't think it's a coincidence that the Kingston SSD came very close to the performance of the Crucial SSD, as it's the most 'expensive' of the cheap drives.

In the end, my intend was to demonstrate with actual benchmarks how cheap SSDs show bad sustained write performance and I think I succeeded. I hope it helps people to understand that good SSD write performance is not a given, especially for cheaper drives.

The Hacker News discussion of this blog post can be found here

Disclaimer

I'm not sponsored in any way. All mentioned products have been bought with my own money.

The graphs are created with fio-plot, a tool I've made and maintain. The benchmarks have been performed with bench-fio, a tool included with fio-plot, to automate benchmarking with Fio.

As I don't have a test system with NVMe, I had to use SATA-based SSDs. The fact that the SATA interface was not the limiting factor in any of the tests, is foreboding. ↩
As a general note, I think the vast majority of users should not worry about SSD longevity in general. Only people with high-volume write workloads should keep an eye on write endurance of SSD and buy a suitable product. ↩
instead of packing the bits really dense together in a cell horizontally, the bits are stacked vertically, saving horizontal space. This allows for higher data densities in the same footprint. ↩
Some SSDs have a static SLC cache, but others size the SLC cache in accordance to how full an SSD is. When the SSD starts to fill up, the SLC cache size is reduced. ↩
After around 45-50 minutes of testing, performance of the Crucial MX 500 also started to drop to around 40 MB/s and fluctuate up and down. Evidence. ↩
it's so funny to me that a hard drive beats an SSD on latency. ↩

How to setup a local or private Ubuntu Mirror

2023-01-18T12:00:00+01:00

Preface

In this article I provide instructions on how to setup a local Ubuntu mirror using debmirror. To set expectations: the mirror will work as intended and distribute packages and updates, but a do-release upgrade from one major version of Ubuntu to the next won't work.

Introduction

By default, Ubuntu systems get their updates straight from the internet at archive.ubuntu.com. In an environment with lots of Ubuntu systems (servers and/or desktops) this can cause a lot of internet traffic as each system needs to download the same updates.

In an environment like this, it would be more efficient if one system would download all Ubuntu updates just once and distribute them to the clients. In this case, updates are distributed using the local network, removing any strain on the internet link¹.

We call such a system a local mirror and it's just a web server with sufficient storage to hold the Ubuntu archive (or part of it). A local mirror is especially relevant for sites with limited internet bandwidth, but there are some extra benefits.

To sum up the main benefits:

Reduced internet bandwidth usage
Faster update proces using the local network (often faster than internet link)
Update or install systems even during internet or upstream outage

The main drawbacks of a local mirror are:

An extra service to maintain and monitor
Storage requirement: starts at 1TB
Initial sync can take a long time depending on internet speed

Mirror solutions

Ubuntu mirror script

This solution is geared towards ISPs or companies who like to run their own regional mirror. It is meant to mirror the entire, unfiltered Ubuntu package archive.

As of 2023 you should expect 2.5TB for archive.ubuntu.com and also around 2.5 TB for ports.ubuntu.com (ARM/RISCV and others).

This is a lot of storage and likely not what most environments need. Even so, if this is what you want to run you can consult this web page and use the script mentioned here.

debmirror

Based on my own research, it seems that the tool Debmirror is the most simple and straight-forward way to create a local Ubuntu mirror with a reasonable data footprint of about 480 GB (2023) for both Jammy AMD64 (22.04) and Focal AMD64 (20.04).

Based on on your needs, you can further finetune Debmirror to only download the pacakges that you need for your environment.

apt-cacher-ng

The tool apt-cacher-ng acts as a caching proxy and only stores updates that are requested by clients. Missing or new updates are only downloaded once the first client requests this download, although there seem to be option to pre-download updates.

Although I expect a significantly smaller footprint than debmirror, I could not find any information about actual real-life disk usage.

Creating an Ubuntu mirror with debmirror

Although apt-cacher-ng is quite a capable solution which many additional features, I feel that a simple mirror solution like debmirror is extremely simple to setup and maintain. This article will this focus on debmirror.

Preparation

1 - Computer

First of all we need a computer - which can be either physical or virtual - that can act as the local mirror. I've used a Raspberry Pi 4B+ as a mirror with an external USB hard drive and it can saturate a local 1 Gbit network with ease.

2 - 1TB storage capacity (minimum)

I'm mirroring Ubuntu 22.04 and 20.04 for AMD64 architecture and that uses around 480 GB (2023). For ARM64, you should expect a similar storage footprint. There should be some space available for future growth so that's why I recommend to have at least 1 TB of space available.

Aside from capacity, you should also think about the importance of redundancy: what if the mirror storage device dies and you have to redownload all data? Would this impact be worth the investment in redundancy / RAID?

It might even be interesting to use a filesystem (layer) like ZFS or LVM that support snapshots to quickly restore the mirror to a known good state if there has been an issue with a recent sync.

3 - Select a local public Ubuntu archive

It's best to sync your local mirror with a public Ubuntu archive close to your physical location. This provides the best internet performance and you also reduce the strain on the global archive. Use the linked mirror list to pick the best mirror for your location.

In my case, I used nl.archive.ubuntu.com as I'm based in The Netherlands.

Ubuntu Mirror configuration

01 - Add the storage device / volume to the fstab

If you haven't done so already, make sure you create a directory as a mountpoint for the storage we will use for the mirror.

In my case I've created the /mirror directory...

mkdir /mirror

... and updated the fstab like this (example!):

/dev/disk/by-uuid/154d28fb-83d0-4848-ac1d-da1420252422 /mirror xfs noatime 0 0

I recommend using the by-uuid or by-id path for mounting the storage device as it's most stable and don't forget the use the correct filesystem (xfs/ext4).

Now we can issue:

mount /mirror

02 - Install required software

We need a webserver installed on the mirror to serve the deb packages to the clients. Installation is straightforward and no further configuration is required. In this example I'm using Apache2 but you can use any webserver you're comfortable with.

If you want to synchronise with the upstream mirror using regular HTTP you don't need additional software.

apt-get update
apt install apache2 debmirror gnupg xz-utils

I think that using rsync for synchronisation is more efficient and faster but you have to configure your firewall to allow outbound traffic to TCP port 873 (which is outside the scope of this tutorial)

apt install rsync

Tip: make sure you run debmirror on a 20.04 or 22.04 system as older versions don't support current ubuntu mirrors and some required files won't be downloaded.

03 - Creating file paths

I've created this directory structure to host my local mirror repos.

/mirror/
├── debmirror
│   ├── amd64
│   │   ├── dists
│   │   ├── pool
│   │   └── project
│   └── mirrorkeyring
└── scripts


mkdir /mirror/debmirror
mkdir /mirror/debmirror/amd64
mkdir /mirror/debmirror/mirrorkeyring
mkdir /mirror/scripts

The folders within the amd64 directory will be created by debmirror so they don't have to be created in advance.

04 - install GPG keyring

gpg --no-default-keyring --keyring /mirror/debmirror/mirrorkeyring/trustedkeys.gpg --import /usr/share/keyrings/ubuntu-archive-keyring.gpg

05 - Create symlinks

We need to create symlinks in the apache2 /var/www/html directory that point to our mirror like this:

cd /var/www/html
ln -s /mirror/debmirror/amd64 ubuntu

06 - Configure debmirror

Debmirror is just a command-line tool that takes a lot of arguments. If we want to run this tool daily to keep our local mirror in sync, it's best to use a wrapper script that can be called by cron.

Such a wrapper script is provided by this page and I have included my own customised version here.

You can download this script and place it in /mirror/scripts like this:

cd /mirror/scripts
wget https://louwrentius.com/files/debmirroramd64.sh.txt -O debmirroramd64.sh 
chmod +x debmirroramd64.sh

Now we need to edit this script and change some parameters to your specific requirements. The changes I've made as compared to the example are:

export GNUPGHOME=/mirror/debmirror/mirrorkeyring
release=focal,focal-security,focal-updates,focal-backports,jammy,jammy-security,jammy-updates,jammy-backports
server=nl.archive.ubuntu.com
proto=rsync
outPath=/mirror/debmirror/amd64

The Ubuntu installer ISO for 20.04 and 22.04 seem to require the -backports releases too so those are included.

Limitations I've not been able (yet) to make the do-release-upgrade process work to upgrade a system from focal to jammy. I found this old resource but those instructions don't seem to work for me.

07 - Limiting bandwidth

The script by default doesn't provide a way to limit rsync bandwidth usage. In my script, I've added some lines to make bandwidth limiting work as an option.

A new variable is added that must be uncommented and can be set to the desired limit. In this case 1000 means 1000 Kilobytes per second.

bwlimit=1000

You also need to uncomment this line:

--rsync-options "-aIL --partial --bwlimit=$bwlimit" \

08 - Initial sync

It may be advised not to first run the initial sync before we configure a periodic cron job to do a daily sync. The first sync can take a long time and may interfere with the cron job. It may be advised to only enable the cronjob once the initial sync is completed.

As the initial sync can take a while, I like to run this job with screen. If you accidentally close the terminal, the rsync process isn't interrupted (although this isnot a big deal if that happens, it just continues where it left off).

apt install screen
screen /mirror/scripts/debmirroramd64.sh

09 - Setup cron job

When the initial sync is completed we can configure the cron job to sync periodically.

0 1 * * * /mirror/scripts/debmirroramd64.sh

In this case the sync runs daily at 1 AM.

The mirror includes all security updates so depending on your environment, it's recommended to synchronise the mirror at least daily.

10 - Client configuration

All clients should point to your local mirror in their /etc/apt/sources.list file. You can use the IP-address of your mirror, but if you run a local DNS, it's not much effort to setup a DNS-record like mirror.your.domain and have all clients reconfigured to connect to the domain name.

This is the /etc/apt/sources.list for the client

deb http://mirror.your.domain/ubuntu RELEASE main restricted universe multiverse
deb http://mirror.your.domain/ubuntu RELEASE-security main restricted universe multiverse
deb http://mirror.your.domain/ubuntu RELEASE-updates main restricted universe multiverse

The RELEASE value should be changed to the appropriate ubuntu release, like bionic, focal or jammy.

If you have an environment with a lot of Ubuntu systems, this configuration is likely provisioned with tools like ansible.

11 - Monitoring

Although system monitoring is out-of-scope for this blog post, there are two topics to monitor:

disk space usage (alert if space is running out)
succesfull synchronisation script execution (alert if script fails)

If you don't monitor the synchronisation process, the mirror will become out-dated and will lack the latest security updates.

Closing words

As many environments are either cloud-native or moving towars a cloud-environment, running a local mirror seems less and less relevant. Yet there may still be environments that could benefit from a local mirror setup. Maybe this instruction is helpful.

You may notice that cloud provides actually also run their own Ubuntu archive mirror to reduce the load on their upstream and peering links. When you deploy a standard virtual machine based on Ubuntu, it is by default configured to use the local mirror. ↩

I resurrected my Dutch movie review site from 2003

2022-06-09T12:00:00+02:00

Introduction

Between 2003 and 2006, I ran a Dutch movie review site called moevie.nl.¹ I built the site and wrote the reviews. It never made any money. It cost me money to host, and it cost me a lot of time writing reviews, but I remember enjoying writing reviews about films I liked.

The gimmick of the site was that the reviews had two parts. The first part is spoiler-free, just giving a recommendation with some context to make up your own mind. The second part contained a reflection of the movie, which included spoilers.

Even back then, the site didn't win any design awards (from archive.org - click to enlarge)²

I started building the site a few months after finishing college (IT) in 2002 as I felt inept and had little confidence. Building something tangible felt like a good way to build up and demonstrate skills. And I had something to say about movies.

Although moevie.nl did not help me gain employment as far as I know, it was fun while it lasted. At some point, I didn't get much joy out of writing movie reviews and I let the site die.

I did keep backups of the database the code and the pictures though. And now after 18+ years I decided to resurrect the site, including all (old) reviews.

Why resurrect a dead website gone for 16+ years?

Rebuilding the site was just a way to spend time, a small hobby project. Something to be bussy with. The second reason is some kind of misplaced nostalgia. I sometimes regret shutting down the site, wondering what could have been if I persevered.

Losing and regaining the domain

Back in 2006, my hosting provider (non-profit with just a few servers) abruptly stopped operating due to hardware failure ³ and I was forced to move my domain to another company. At that time, private citizens in The Netherlands could not register an .nl domain, only businesses could, so that was a bit of a hassle.

Not long thereafter however, I decided to let the domain expire. It was quickly scooped up by 'domain resellers'. Years later I decided that I wanted moevie.nl back, but the sellers always asked insane amounts of money.

In 2019, I visited moevie.nl on a whim. To my surprise it didn't resolve anymore, the domain was available! I quickly scooped it up, but I didn't do much with it for a long time, until now.

Rebuilding the site

I really wanted to preserve the aesthetic of moevie.nl as it was back then. Especially in the context of modern web design, it does stand out. As a sore thumb - but still - I had a goal.

Having the code and database dump is one thing, but it doesn't tell you what it actually looked like in 2003-2006. I could have tried to get the old (PHP4) code working, but I just didn't feel like it.

Instead, I chose to visit Archive.org and indeed, it captured old snapshots of my site back in 2006. So those were of great help. The screenshots at the top of this blog post are lifted from this page on archive.org. This snapshot was taken just before I decided to close the site.

The challenge of mobile device screens

To set the stage a bit: the rise and fall of moevie.nl happened a year before the iPhone was first announced. Smartphones from Blackberry were popular. I had a Palm VX PDA and later a HP Compaq PDA.

Most people didn't have mobile data connections so as far as I know, the mobile web really wasn't a thing yet.

So moevie.nl was primarily developed for the desktop. When I thought I was finished rebuilding the site, I quickly discovered that the site was unusable on my iPhone and way too small and finicky to use on my iPad.

For somebody who has no experience with modern web development, it was quite a steep learning-curve discovering how to deal with the various screen sizes in CSS⁴.

A very large part of the entire effort of rebuilding the site was spend on making the site workable on all different device sizes. Fortunately, iOS device simulators were of great help on that front.

Technology

I've recreated moevie.nl with Python and Django. For the database, I chose Postgresql, although that is total overkill, I could have used SQLite without any issues.

I chose Django because I'm quite familiar with Python so that was a straight-forward choice. I selected Postgresql mostly just to regain some knowledge about it.

Hosting

I'm self-hosting moevie.nl on the same Raspbery Pi4 that is hosting this blog. This Raspberry Pi is powered by the sun.

So moevie.nl is solar-powered during the day and battery-powered during the night.

Closing words

I'm not sure if I really want to start writing movie reviews again, knowing full well how much effort it takes. Also I'm not sure I have anything to say about movies anymore, but we'll see.

The overall experience of rebuilding the site was frustrating at times due to the severe lack of experience and knowledge. Now that the site is done and working, even on mobile devices, that feels good.

The name is based on the phonetic pronunciation in Dutch of the English word 'movie'. ↩
sorry for the language but I could not find a better screenshot. ↩
I was neglecting the site at that time due to losing motivation. ↩
I admit I only tested with iOS devices so Android-based smartphones could experience issues. ↩

An ode to the 10,000 RPM Western Digital (Veloci)Raptor

2021-10-30T12:00:00+02:00

Introduction

Back in 2004, I visited a now bankrupt Dutch computer store called MyCom¹, located at the Kinkerstraat in Amsterdam. I was there to buy a Western Digital Raptor model WD740, with 74 GB of capacity, running at 10,000 RPM.

When I bought this drive, we were still in the middle of the transition from the PATA interface to SATA². My raptor hard drive still had a molex connector because older computer power supplies didn't have SATA power connectors.

You may notice that I eventually managed to break off the plastic tab of the SATA power connector. Fortunately, I could still power the drive through the Molex connector.

A later version of the same drive came with the Molex connector disabled, as you can see below.

Why did the Raptor matter so much?

I was very eager to get this drive as it was quite a bit faster than any consumer drive on the market at that time.

This drive not only made your computer start up faster, but it made it much more responsive. At least, it really felt like that to me at the time.

The faster spinning drive wasn't so much about more throughput in MB/s - although that improved too - it was all about reduced latency.

A drive that spins faster³ can complete more I/O operations per second or IOPs⁴. It can do more work in the same amount of time, because each operation takes less time, compared to slower turning drives.

The Raptor - mostly focussed on desktop applications⁵ - brought a lot of relief for professionals and consumer enthusiasts alike. Hard disk performance, notably latency, was one of the big performance bottlenecks at the time.

For the vast majority of consumers or employees this bottleneck would start to be alleviated only well after 2010 when SSDs slowly started to become standard in new computers.

And that's mostly also the point of SSDs: their I/O operations are measured in micro seconds instead of milliseconds. It's not that throughput (MB/s) doesn't matter, but for most interactive applications, you care about latency. That's what makes an old computer feel as new when you swap out the hard drive for an SSD.

The Raptor as a boot drive

For consumers and enthusiast, the Raptor was an amazing boot drive. The 74 GB model was large enough to hold the operating system and applications. The bulk of the data would still be stored on a second hard drive either also connected through SATA or even still through PATA.

Running your computer with a Raptor for the boot drive, resulted in lower boot times and application load times. But most of all, the system felt more responsive.

And despite the 10,000 RPM speed of the platters, it wasn't that much louder than regular drives at the time.⁷.

In the video above, a Raspberry Pi4 boots from a 74 GB Raptor hard drive.

Alternatives to the raptor at that time

To put things into perspective, 10,000 RPM drives were quite common even in 2003/2004 for usage in servers. The server-oriented drives used the SCSI interface/protocol which was incompatible with the on-board IDE/SATA controllers.

Some enthusiasts - who had the means to do so - did buy both the controller⁸ and one or more SCSI 'server' drives to increase the performance of their computer. They could even get 15,000 RPM hard drives! These drives however, were extremely loud and had even less capacity.

The Raptor did perform remarkably well in almost all circumstances, especially those who mattered to consumers and consumer enthusiasts alike. Suddenly you could get SCSI/Server performance for consumer prices.

The in-depth review of the WD740 by Techreport really shows how significant the raptor was.

The Velociraptor

The Raptor eventually got replaced with the Velociraptor. The Velociraptor had a 2.5" formfactor, but it was much thicker than a regular 2.5" laptop drive. Because it spun at 10,000 RPM, the drive would get hot and thus it was mounted in an 'icepack' to disipate the generated heat. This gave the Velociraptor a 3.5" formfactor, just like the older Raptor drives.

In the video below, a Raspberry Pi4 boots from a 500 GB Velociraptor hard drive.

Benchmarking the (Veloci)raptor

Hard drives do well with sequential read/write patterns, but their performance implodes when the data access pattern becomes random. This is due to the mechanical nature of the device. That random access pattern is where 10,000 RPM outperform their slower turning siblings.

Random 4K read performance showing both IOPs and latency. This is kind of a worst-case benchmark to understand the raw I/O and latency performance of a drive.

Drive ID	Form Factor	RPM	Size (GB)	Description
ST9500423AS	2.5"	7200	500	Seagate laptop hard drive
WD740GD-75FLA1	3.5"	10,000	74	Western Digital Raptor WD740
SAMSUNG HD103UJ	3.5"	7200	1000	Samsung Spintpoint F1
WDC WD5000HHTZ	2.5" in 3.5"	10,000	500	Western Digital Velociraptor
ST2000DM008	3.5"	7200	2000	Seagate 3.5" 2TB drive
MB1000GCWCV	3.5"	7200	1000	HP Branded Seagate 1 TB drive

I've tested the drives on an IBM M1015 SATA RAID card flashed to IT mode (HBA mode, no RAID firmware). The image is generated with fio-plot, which also comes with a tool to run the fio benchmarks.

It is quite clear that both 10,000 RPM drives outperform all 7200 rpm drives, as expected.

If we compare the original 3.5" Raptor to the 2.5" Velociraptor, the performance increase is significant: 22% more IOPs and 18% lower latency. I think that performance increase is due to a combination of the higher data density, the smaller size (r/w head is faster in the spot it needs to be) and maybe better firmware.

Both the laptop and desktop Seagate drives seem to be a bit slower than they should be based on theory. The opposite is true for the HP (rebranded Seagate), which seem to perform better than expected for the capacity and rotational speed. I have no idea why that is. I can only speculate that because the HP drive came out of a server, that the fireware was tuned for server usage patterns.

Closing words

Although the performance increase of the (veloci)raptor was quite significant, it never gained wide-spread adoption. Especially when the Raptor first came to marked, its primary role was that of a boot drive because of its small capacity. You still needed a second drive for your data. So the increase in performance came at a significant extra cost.

The Raptor and Velociraptor are now obsolete. You can get a solid state drive for $20 to $40 and even those budget-oriented SSDs will outperform a (Veloci)raptor many times over.

If you are interested in more pictures and details, take a look at this article.

This article was discussed on Hacker News here.

Reddit thread about this article can be found here

Mycom, a chain store with quite a few shops in all major cities in The Netherlands, went bankrupt twice, once in 2015 and finally in 2019. ↩
We are talking about the first SATA version, with a maximum bandwidth capacity of 150 MB/s. Plenty enough for hard drives at that time. ↩
https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics ↩
https://louwrentius.com/understanding-storage-performance-iops-and-latency.html ↩
I read that WD intended the first Raptor (34 GB version) to be used in low-end servers as a cheaper alternative to SCSI drives . After the adoption of the Raptor by computer enthusiasts and professionals, it seems that Western Digital pivoted, so the next version - the 74 GB I have - was geared more towards desktop usage. That also meant that this 74 GB model got fluid bearings, making it quieter⁶. ↩
The 74 GB model is actually rather quiet drive at idle. Drive activity sounds rather smooth and pleasant, no rattling. ↩
Please note that the first model, the 37 GB version, used ball bearings in stead of fluid bearings, and was reported to be significant louder. ↩
Low-end SCSI card were often used to power flatbed scanners, Iomega ZIP drives, tape drives or other peripherals, but in order to benefit from the performance of those server hard drives, you needed a SCSI controller supporting higher bandwidth and those were more expensive. ↩

Raspberry Pi as a router using a single network interface

2021-09-29T12:00:00+02:00

Introduction

Disclaimer: this article is intended for consumers and hobbyists.

If you want to run your own router at home, the Raspberry Pi 4 Model B¹ can be an excelent hardware choice:

it's fairly cheap
it's fast enough
it can saturate it's gigabit network port
it is power-efficient

The key problem it seems, is that it has only one, single network interface. If you build a router, you need at least two:

The first interface connected to your internet modem/router (ideally in bridge mode)
The second interface connected to your home network (probably a switch)

So if you would use the Raspberry Pi, you would probably buy a gigabit USB3 NIC for around $20 and be done with it.

click on the image for a larger version

Now, what if I told you that you can build exactly the same setup by using only the single on-board network interface of the Raspberry Pi 4?

How is that possible?

Introducing VLANs

Yes, I'm introducing the reader to a technology that exists since the '90s. It is widely used within businesses and other organisations.

Because I have a sense that this technology is less well-known in circles outside IT operations, I think it may be an interesting topic to discuss.

Understanding VLANs

VLAN technology allows you to run different, separate networks over the same, single physical wire and on the same, single switch. This saves a lot on network cabling and the number of physical switches required if you want to operate networks that are separate from each other.

If you want to run traffic from different networks over the same physical wire or switch, how can you identify those different traffic flows?

With VLAN technology enabled, such network 'packets' are labeled with a tag. As the VLAN technology operates at the level of Ethernet, we should not talk about 'packets' but about 'ethernet frames'. The terminology is not important to understand the concept, I think.

It suffices to understand that there is a tag put in front of the ethernet frame, that tells any device that supports VLANs to which network a frame and thus a packet belongs.

This way, network traffic flows, can be distinguished from each other. And those tags are nothing fancy, they are called a VLAN ID and it is just a number between 1 and 4096².

Managed switch

Now that we understand the concept of VLANs, how do we use it?

First of all, you need a managed network switch that supports VLANs.

The cheapest switch with VLAN support I could find is the TP-LINK TL-SG105E, for around 25 euros or dollars. This is a 5-port switch, but the 8-port version is often only a few euros/dolars more.

Juan Pedro Paredes in the comments point out that this TP-LINK switch may not be able to handle the large number of ARP requests that may arrive at the port connected to the Internet Modem. Others are quite negative about this switch in the Hacker News discussion (linked below). I'm not sure if Netgear switches, which are near the same price, fare any better.

A switch like this has a web-based management interface that allows you to configure VLANS on the device.

Tagged vs untagged

In the context of VLANS, a network switch port can be in two states:

Member of a particular network (VLAN) (untagged)
Transporting multiple networks (VLANs) (tagged)

If a port is just a member of a VLAN, it just behaves like any other switch port. In this mode, it can obviously only be a member of one network/VLAN. The VLAN tags are stripped off all network traffic coming out of this port.

However, a port that is assigned 'tagged' VLAN trafic, just forwarded traffic as-is, including their VLAN tag.

This is the trick that we use to send network packets from different networks (VLANS) to our Raspberry Pi router over a single port/wire.

click on the image for a larger version

So let's unpack this picture together, step by step.

Let's imagine a (return) packet from the Internet arrives at the modem and is sent into switchport 1.

The switch knows that any traffic on that switch port belongs to VLAN 10. Since this traffic needs to be send towards the Pi Router, it will put a tag on the packet and forwards the packet, including the tag towards the Pi on switch port 2.

The Pi - in turn - is configured to work with VLANs just as the switch. The tag on the packet tells the Pi to wich virtual interface the packet must be send.

A netplan configuration example to illustrate this setup:

network:
  version: 2
  ethernets:
    enp2s0f0:
      dhcp4: no
  vlans:
    enp2s0f0.10:
       id: 10
       link: enp2s0f0
       addresses:
         - 68.69.70.71/24 (fake internet address)
       gateway4: 68.69.70.1 (fake upstream ISP router)
    enp2s0f0.20:
       id: 20
       link: enp2s0f0
       addresses:
         - 192.168.0.1/24 (internal network address, acting as gateway)

As you can see, the VLAN packets that arrive as tagged packets, are send (without their tags) to a virtual network interface belonging to that particular network. Those virtual network interfaces all share the same physical interface (enp2s0f0). The virtual network interfaces are just the physical interface name with ".(VLAN ID)" added.

From here on out, you probably understand where this is going: those two virtual network interfaces are basically similar to a setup with two physical network interfaces. So all the routing and NAT that needs to happen, just happens on those two virtual interfaces instead.

How to work with VLANs

To work with VLANs, you need a managed switch that supports VLANs. A managed switch has a management interface, often a web-based management interface.

In this example, I'm using the TP-LINK TL-SG105E switch as an example. To get to this page, go to VLAN --> 802.1Q VLAN in the web interface.

So from this table we can derive that:

Port 1 is an untagged member of VLAN 10
Port 2 is a tagged member of VLAN 10 and VLAN 20
Port 3 is an untagged member of VLAN 20

Please note that it is also recommended to remove ports from VLANs they don't use. So I removed ports 1, 2 and 3 from the default VLAN 1.

Now, if you have more devices to connect to the internal LAN on this switch, you need to configure the ports to be an untagged member of VLAN 20.

Caveats

Bandwidth impact

Obviously, if you use a single interface, you only get to use the bandwidth of that sinle interface. In most cases, this is not an issue, as gigabit ethernet is full-duplex: there is physical exclusive wiring for upstream traffic and downstream traffic.

So you might say that full-duplex gigabit ethernet has a raw throughput capacity of two gigabit/s, although we mostly don't talk about it that way.

So when you download at 200 Mbit/s, that traffic is ingested over VLAN 10 over the incomming traffic path. It is then sent out over VLAN 20 towards your computer over VLAN 20 using the outgoing path. No problem there.

If you would also use the Raspberry Pi as a backup server (with an attached external hard drive), the backup traffic and the internet traffic could both 'fight' for bandwidth on the same gigabit link.

Impact on gigabit internet

Update June 2022 I was actually able to use full Gigabit internet speed over VLANs, at around 111 MB/s. I made some mistakes during earlier testing.

You will never get the full gigabit internet network speed if you would build this setup. It will probably max out at ~900 Mbit. (I'm assuming here that you would use x86 hardware as the Pi would not be able to handle firewalling this traffic anyway.)

This is because most traffic is based on TCP connections and when you download, there is traffic both ways!. The download traffic is the bulk of the traffic, but there is a substantial steady stream of return packets that acknowledges to the sender that traffic has been received (if not, it would trigger a retransmission).

Remember that in this single-port setup, the Pi uses the same gigabit port to send the return traffic to the internet over VLAN 10 and the download data towards your home computer over VLAN 20. So the size of the upstream traffic will limit your maximum download performance.

The Raspberry Pi 4 Model B as a router

The biggest limitation - which becomes an issue for more and more people - is performance. If you use IPTABLES on Linux for firewalling, in my experience, network throughput drops to a maximum of 650 Mbit/s.

That's only an issue (first world problems) if you have gigabit internet or an internet speed beyond what the Pi can handle.

If your internet speed doesn't even come close, this is not an issue at all.

Maybe the Raspberry Pi 400 or the compute module performs better in this regard as their CPUs are clocked at higher Ghz.

Closing words

If it makes any sense for you to implement this setup, is only for you to decide. I'm running this kind of setup (using an x86 server) for 10 years as I can't run a second cable from my modem to the room where my router lives. For a more detailed picture of my home network setup, take a look here.

Feel free to leave any questions of comments below.

The hacker news discussion about this article can be found here.

Route-on-a-stick

I learned from the hacker news discussion that a router with just one network interface is called a router on a stick.

Older models of the Raspberry Pi are significantly network bandwidth constrained. So much so, that they would not be suitable as Internet routers if your internet speed is above 100Mbit. ↩
most cheap switches can't operate more than 32 ~ 64 VLANs maximum. Only more expensive, enterprise gear can work with the full 4096 VLANS at the same time. However, this is probably not relevant for consumers. ↩

A practical understanding of lead acid batteries

2021-08-29T12:00:00+02:00

Introduction

The goal of this article is to give you a practical understanding Lead Acid batteries. We won't address the underlying chemistry, we'll treat them as a black-box and we will discover their characteristics and how to keep them healthy.

Disclaimer

I'm an amateur. I have absolutely zero relevant background in battery technology or electronics. I just scraped some information together in a hopefully useful manner.

A high-level overview of the lead acid battery

It can provide a ton of current / power
It hates to be deep-discharged and will die quickly if done repeatedly
It hates being in a discharged state
Only use 50% of total capacity if longevity matters (ideally only 30%)
It's usable capacity depends on the load
They are slow to charge (8-12 hours)
They don't perform as well in cold weather

Lead acid batteries can provide a lot of current

Lead acid batteries can put out so much current that you can use them to weld². They are widely used in ICE cars to power the starter motor, which needs hundreds of amps at 12 volt to turn over the engine.

They are also used to power mobility scooters, golf carts, trolly motors, small toy cars for children to ride in, or provide electricity on boats, caravans and in RVs. You can also find them in more stationary applications such in UPS systems¹ or - of course - solar battery banks.

Danger

Lead acid batteries typically don't have any kind of short-circuit protection build-in. This means that if you (accidentally) short-circuit a lead acid battery, the battery can explode or it can cause a fire. Whatever object caused the short-circuit, will probably be destroyed.

Because lead acid batteries can supply such high currents, it's important to assure that you use the right wire thickness / diameter. If the wire is too thin, it causes too much resistance and thus may overheat, causing the insulation to catch fire.

Lead acid batteries can be very dangerous, so you have to be very carefull with them. Personally, I always make sure that anything connected to a lead acid battery is properly fused.

Lead acid batteries hate being deep discharged

The common rule of thumb is that a lead acid battery should not be discharged below 50% of capacity, or ideally not beyond 70% of capacity. This is because lead acid batteries age / wear out faster if you deep discharge them.

The most important lesson here is this:

Although a lead acid battery may have a stated capacity of 100Ah, it's practical usable capacity is only 50Ah or even just 30Ah

If you buy a lead acid battery for a particular application, you probably expect a certain lifetime from it, probably in years. If the battery won't last this long, it may not be an economically viable solution.

image source - Please note that this chart is based on a heavy-duty lead acid battery and doesn't reflect the lifecycle of a regular consumer lead acid battery. It is advised to look up the relevant chart for the particular battery model you may be interested in buying.

If you cycle a battery (with the characteristics depicted in the chart) every day as part of some kind of off-grid solar setup and you use 80% of it's capacity, you'll probably have to replace it after about two years.

If you add a few extra batteries in parallel, individual batteries may only be used 20% to 30% of capacity, and those same batteries may last 6 - 9 years. So by spending 2 or 3 times the money on batteries, you get 3 to 4 times the lifetime out of your setup.

So, for example, if you really need 100Ah of battery capacity, you may need two 100Ah batteries in parallel to assure longevity. You even may decide to buy three 100Ah batteries just to assure that they will last for the desired number of cycles.

However, if the battery setup is only meant for emergency power and thus only expected to operate a few times a year, discharging a lead acid battery to 80% of capacity is not a big deal. There is no need to add extra battery capacity because the number of charge/discharge cycles is so low that there isn't that much wear on the battery.

Lead acid batteries eventually die from old age

A lead acid battery deteriorates just by ageing. So even if it's kept full charged most of the time, it will wear out and needs to be replaced after a few years. It doesn't matter how well you treat them, even with the best care, they need to be replaced eventually.

Lead acid batteries hate being in a discharged state

Lead acid batteries should never stay discharged for a long time, ideally not longer than a day. It's best to immediately charge a lead acid battery after a (partial) discharge to keep them from quickly deteriorating.

A battery that is in a discharged state for a long time (many months) will probably never recover or ever be usable again even if it was new and/or hasn't been used much.

Usable capacity depends on the load

A typical 12-volt battery has a rating stated in ampere hour that tells you the capacity. For example, a battery can be rated as 70Ah.

So this could mean that the battery can sustain a load of 7A for 10 hours or 70A for one hour, right?

Unfortunately, no

It turns out that the usable capacity of a lead acid battery depends on the applied load. Therefore, the stated capacity is actually the capacity at a certain load that would deplete the battery in 20 hours.

This is concept of the C-rate. 1C is the theoretical one hour discharge rate based on the capacity. Batteries are mostly sold with a capacity based on a 0.05C discharge rate for 20 hours.

The C-rate is important because the C-rate is related to the usable capacity of a battery. That 70Ah capacity rating is based on a 0.05 C-rate or 20-hour discharge rate. That would be 70Ah / 20 = 3.5A.

This is important to understand: if you would put a higher load on this battery, the usable capacity will be less than 70Ah. For example, with a 7A load, the usable capacity may only be 64Ah (fake number for illustration purposes).

It also works in your favor: if the load is less than the 0.05 C-rate, the actual usable capacity will be higher!

So why is this?

When you put a load on a battery, the voltage drops a bit. Higher loads cause larger voltage drops, or to put it differently: the battery 'struggles' to maintain voltage.

Image source

So if a load exceeds the standard 0.05C rate (C/20), you may have to select a higher capacity battery or accept a shorter run-time than you might expect based on the rated capacity on the label.

You even may consider putting multiple batteries in parallel to reach the desired usable capacity / runtime.

WARNING

The chart about the state-of-charge under load shows that you should keep an eye on the actual load and voltage. With a C/20 load, the battery is at 50% at 12.30 volt³.

A C/5 load on a 70Ah battery would be 14A. At that load, the battery is at 50% capacity at ~11.55 Volt under load. Only the load in combination with the voltage may give an indication of actual state-of-charge.

Predicting state-of-charge under load is doable with a static, constant load, but becomes more difficult when the load fluctuates, so take this into account.

ANOTHER WARNING

Different manufacturers produce different batteries that may have different discharge characteristics. This means that you should look up the battery specifications and hopefully find a discharge rate chart that will help you gauge actual capacity under load for this particular model.

How do you know the state of charge of a lead acid battery?

The state of charge is measured at rest: when the battery is not connected to any load or charger for 24 hours. The voltage will reflect the state of charge (SoC).

WARNING

There are many different, conflicting tables to be found on the internet that correlate voltage with a particular state of charge. Be sure you check that you pick the right one, consult the footnote⁴ for more information.

State of Charge (SoC)	Voltage at rest (24h)
100%	12.70+
75%	12.40
50%	12.20
25%	12.00
0%	11.80

Please note that this table is only valid at an ambient temperature of 25C / 77F. If the temperature is lower, usable capacity diminishes and the voltages at wich a certain SoC is reached, will be higher.

Furthermore, these numbers can deviate a little bit depending on the kind of lead acid battery.

If you measure the voltage under load - for example, when you power some lights - the voltage does not reflect the actual state of charge.

It is quite difficult to determine the state of charge under load. Sometimes, battery manufactures provide a discharge chart that allows you to determine the state-of-charge based on the current load.

But often it is something you have to measure or figure out yourself. A constant load makes estimating battery capacity under load more predictable, but if the load varies, it is more difficult to accurately gauge the state of charge.

The positive impact on capacity of connecting batteries in parallel

By using multiple batteries in parallel, the load is also shared across all batteries. Each individual battery only has to supply a fraction of the total load. This means that in addition to the extra usable capacity of the added batteries, there is also added usable capacity because of the reduced load on each individual battery.

For example, if a 100Ah battery has a 0.05C discharge rate of 5A. If it has to provide 10A, the usable capacity is lower than the advertised 100Ah as explained earlier. If we add a second 100A battery in parallel, each battery now needs to supply only half of the load and thus will be able to provide the stated capacity as it is precisely the 0.05C discharge rate.

Lead acid batteries need deep discharge protection

It is highly recommended to use lead acid batteries in combination with a low-voltage cut-off solution that protects the battery against deep discharge⁵.

this article is not sponsored by victron

Ideally you can configure the cut-off coltage, such as with the depicted unit.

So many lead acid batteries are 'murdered' because they are left connected (accidentally) to a power 'drain'.

Charging a lead acid battery

No matter the size, lead acid batteries are relatively slow to charge. It may take around 8 - 12 hours to fully charge a battery from fully depleted. It's not possible to just dump a lot of current into them and charge them quickly. That would just overload and destroy the battery⁸.

Lead acid batteries need a specific 3-stage charge process⁶ in order to preserve their condition.

In practice, if you don't discharge a battery beyond 50%, it takes less time to recharge the battery⁷.

It can be a good idea to hookup unused batteries permanently to a 'tricklecharger'. This is a charger that charges the battery with a maximum current of 0.8A.

As it can take a very long time to charge a larger capacity battery with a tricklecharger, you need a regular charger, that can supply a decent current, to charge a battery 'within a reasonable timeframe'.

Lead acid battery types

Flooded / FLA

This is the well-known older type of battery. It may be necessary to add distilled water from time to time, so they require maintenance.

The key problem with batteries that require maintenance is that most people (consumers) don't know and if they know, they forget. These batteries basically don't match well with 'human nature'.

It seems to me that these batteries are on their way out in the consumer space, but are still prevalent in commercial/industrial application. It's probably easy for a business to just have a trained employee or service company periodically maintain the batteries.

EFB or Enhanced Flooded Battery

These batteries are improved versions of the regular flooded battery. They are more expensive, but will last more charge/discharge cycles, especially with deeper discharges.

Although not as performant as AGM batteries (which will be discussed shortly), they provide a cheaper alternative to AGM batteries.

Sealed Lead Acid

This type of battery is fully sealed. SLA batteries essentially the same as VRLA batteries but this name is used for the smaller capacity batteries, as found in motorcycles, uninterruptible power supplies and such.

These are maintenance-free batteries. They never require any maintenance during their lifetime. You don't need to add distilled water or anything during their lifetime.

Valve-Regulated Lead Acid

This name is used for batteries like the SLA battery, but with higher capacities. See also wikipedia. They have liquid inside like the flooded battery, but they are sealed and don't need any maintenance. To be precise: they can't be maintained, only be replaced.

The 'valve(s)' are only there in case of emergency, to release pressure due to gas buildup within the battery case if charged incorrectly.

AGM (Absorbent Glass Mat)

This is also a fully sealed SLA/VRLA battery, but it is even more advanced. They are better able to withstand deep discharges and can be recharged faster. This comes at a relatively steep price.

The faster recharge cycle can be important if used within a solar power bank, because there are only a limited number of hours when the sun provides enough energy for charging.

Deep-Cycle

These batteries are build differently⁹ and are less suited for starting cars, but better suited to provide power to power boats, RC vans or form a solar power bank.

They are often not a kind of battery in and of itself: there are just regular flooded deep-cycle batteries, or AGM deep-cycle batteries. They are often specifically designed for solar power banks or similar applications.

Evaluation

Although regular flooded batteries will have the longest lifespan of all lead acid battery technology, they require regular maintenance and that may not be practical. Therefore, AGM or other maintenance-free batteries are better suited for residential battery applications, the relatively lower life expectancy is just the price for practicality/convenience.

Low self-discharge rate and storing batteries

Lead acid batteries needs to be stored fully charged. They should be recharged at least every six months due to self-discharge, although the self-discharge rate is rather low.

Buyer beware - ask for fresh batteries

I've ordered quite a few smaller SLA batteries from various brands to test their capacities. I noticed that the actual brand didn't matter much. The age of the battery seemed to matter.

some of the tested SLA batteries

While they are in storage at the vendor, they are probably never recharged, which deteriorates the battery. The batteries with a lower SoC correlated with a serial number that indicated that they were older than the other batteries.

So it might be beneficial to specifically ask for a 'fresh' battery when you order a lead acid battery.

Q & A

Can my lead acid battery be revived?

No.

If the voltage of a 12 volt battery at rest is close to zero, it is dead.

There are tips like 'using epsomsalts' or keeping them on a charger for weeks, but at best, you get only a small portion of usable capacity back, if any. A battery 'revived' like this should never power something you rely on. Personally I don't think it's worth the cost of epsom salt or your time, but you have to decide for yourself if that's true or not.

If a battery is totally dead, I would recommend to accept the loss and get a new one.

The impact of cold weather on performance

If a lead acid battery is exposed to colder or even freezing temperatures, it will work fine, but it can output less current. This is relevant for older, more worn-down batteries. Such batteries can still work fine in the summer, but may no longer be able to start a car or provide another utility with sufficient power when temperatures drop significantly.

Does it make sense to use Lead acid batteries for an off-grid solar setup?

You can do a lead acid solar setup if you can get those batteries cheap but otherwise it may be better to go for a LiFePo4 based setup. Although the initial investment is much higher, Lithium-based batteries will be cheaper long-term because they last so much longer than lead acid batteries (life-time).

I think lead acid batteries are suited for climates with a lot of sunlight available all year round, to power a livingspace through the night.

Since lead acid batteries don't 'like' to be in a discharged state for a long time (more than a day at most), I don't think they are suitable for a more temperate climate, with lots of overcast days.

So the first issue with lead acid batteries is that they don't take well being in a discharged state for more than a day or so. It will make them deteriorate faster.

I think the second issue with lead acid batteries as a solar power bank is their slow charging speed. Lead acid batteries often can't use all available solar power to charge because they just can't charge any faster, no matter their capacity.

This means that even though there would have been enough energy available to fully charge the batteries, it was not available long enough to fully charge the batteries. Maybe AGM batteries may help as they can be charged with higher currents, even though they may not last as long.

Lithium-based batteries can be charged with very large currents and can - in some sense - capture every bit of sunlight that's available. This is much better suited to climates with more intermittent sunny days or even sunny hours, I think.

Another thing that comes to mind is that if you really want to go with lead acid batteries for a solar bank, flooded may be the longest lasting, but the regular maintenance they require may quickly become a chore / unmanageable. I have zero experience with this, but please verify this beforehand. All the more reason to consider at least maintenance-free lead acid batteries, even if they may not last as long.

This is just my thought, I'm no expert on this.

Just remember that regular car batteries are just not suitable for this application. You need - more expensive - batteries that are build specifically for being used in a power bank¹⁰.

Why are lead acid batteries so widely used in cars?

Cars need a power source that can provide a lot of power to run the starter motor. Starter motors can use anywhere from 1.5 to 3 Kilowatt when cranking the engine. That's about 125A to 250A of current at 12 volts.

You may notice that batteries are often rated for much higher CCA or 'Cold Cranking Amps' values, but since they deteriorate over time, that extra margin will come in handy. Especially in colder weather.

Lead acid batteries as used in cars can last many years because they are used under near ideal conditions. They are always kept fully charged and are ony briefly and slightly discharged. They are immediately recharged after the car is started.

How can I check if a battery is healthy ?

You need a battery tester for this. They can be had for around 50 Euro's, which is not far off from just buying a new battery, which you might have to do anyway.

A demonstration video of such a cheap charger.

A UPS can be quite small, to power just a single computer, running off a 'small' 12 volt 7Ah lead acid battery (depicted further down below in the acticle). A step up in size would be a 19-inch rackmounted UPS, which can often be expanded with multiple external battery packs. A datacenter scale UPS is build using many large batteries in both series for higher voltages and in parallel for higher capacity. Lead acid batteries are well-suited for these type of applications because they are always kept fully charged and rarely (fully) discharged. In datacenter applications, they often only need to last until the diesel generators kick in. ↩
Just because you can, doesn't mean you should. Don't do it. ↩
Notice the voltages in the C/20 discharge rate - which should reflect the numbers in the table shown earlier - are actually a bit higher. If you want to be safe, using higher voltages is always safer for battery longevity, but at the cost of usable capacity. ↩
This article goes into more detail about this. Be sure you look at a table that correlates resting voltage against SoC and not the voltage under load. If you see a table with 10.8 volts at 0%, you are looking at a table for under load voltages. A battery at 10.5 - 10.8 volts at rest is probably damaged. A lead acid battery should never be below 11.80 volt at rest. ↩
'bad' battery protection solutions will just start to oscillate as the battery voltage recovers (above the cut-off threshold) when the load is removed. I bought a cheap 20 Euro unit and it was effectively useless because of this problem. ↩
https://en.wikipedia.org/wiki/IUoU_battery_charging ↩
If Lithium-based batteries have one big upside over lead acid batteries in energy storage applications, it might be this aspect: they can be charged much faster. It may make sense to oversize the solar power array just to charge the batteries as quickly as possible within the limited number of available 'sun-hours'. ↩
It is critical that a proper battery charger is used. You should never just apply a static current as overcharging the battery may lead to the buildup of flammable gasses like hydrogen. There are many documented cases of car batteries exploding in this way. Not only can you get hurt by debris, the internal liquid is acidic which can cause significant burns and is especially dangerous for the eyes. ↩
They have ticker plates that are better able to withstand deep discharges at the cost of lower peak current. ↩
I myself do use regular car batteries as part of my solar-powered blog because I got them for free and even if they are shot, they may last for quite a bit. I can also imagine that people would actually build a battery bank made of old car batteries and just ad a whole lot of them, if you have the space. I'm not sure if that kind of setup would be quite reliable. ↩
The car batteries are free, and I had no other use for the gel batteries so I hooked those up too (in parallel). The batteries have wildly different capacities and this is absolutely not recommended. If you hook up batteries in parallel, always use the same capacity. ↩

ZFS RAIDZ expansion is awesome but has a small caveat

2021-06-22T12:00:00+02:00

Introduction

Update April 2023: It has been fairly quiet since the announcement of this feature. The Github PR about this feature is rather stale and people are wondering what the status is and what the plans are. Meanwhile, FreeBSD has announced In February 2023 that they suspect to integrate RAIDZ expansion by Q3.

One of my most popular blog articles is this article about the "Hidden Cost of using ZFS for your home NAS". To summarise the key argument of this article:

Expanding ZFS-based storge can be relatively expensive / inefficient.

For example, if you run a ZFS pool based on a single 3-disk RAIDZ vdev (RAID5 equivalent²), the only way to expand a pool is to add another 3-disk RAIDZ vdev¹.

You can't just add a single disk to the existing 3-disk RAIDZ vdev to create a 4-disk RAIDZ vdev because vdevs can't be expanded.

The impact of this limitation is that you have to buy all storage upfront even if you don't need the space for years to come.

Otherwise, by expanding with additional vdevs you lose capacity to parity you may not really want/need, which also limits the maximum usable capacity of your NAS.

RAIDZ vdev expansion

Fortunately, this limitation of ZFS is being addressed!

ZFS founder Matthew Ahrens created a pull request around June 11, 2021 detailing a new ZFS feature that would allow for RAIDZ vdev expansion.

Finally, ZFS users will be able to expand their storage by adding just one single drive at a time. This feature will make it possible to expand storage as-you-go, which is especially of interest to budget conscious home users³.

Jim Salter has written a good article about this on Ars Technica.

There is still a caveat

Existing data will be redistributed or rebalanced over all drives, including the freshly added drive. However, the data that was already stored on the vdev will not be restriped after the vdev is expanded. This means that this data is stored with the older, less efficient parity-to-data ratio.

I think Matthew Ahrends explains it best in his own words:

After the expansion completes, old blocks remain with their old data-to-parity ratio (e.g. 5-wide RAIDZ2, has 3 data to 2 parity), but distributed among the larger set of disks. New blocks will be written with the new data-to-parity ratio (e.g. a 5-wide RAIDZ2 which has been expanded once to 6-wide, has 4 data to 2 parity). However, the RAIDZ vdev's "assumed parity ratio" does not change, so slightly less space than is expected may be reported for newly-written blocks, according to zfs list, df, ls -s, and similar tools.

So, if you add a new drive to a RAIDZ vdev, you'll notice that after expansion, you will have less capacity available than you would theoretically expect.

However, it is even more important to understand that this effect accumulates. This is especially relevant for home users.

I think that the whole concept of starting with a small number of disks and expand-as-you-go is very desirable and typical for home users. But this also means that every time a disk is added to the vdev, existing data is still stored with the old data-to-parity rate.

Imagine that we have a 10-drive chassis and we start out with a 4-drive RAIDZ2.

If we keep adding drives⁵ conform this example, until the chassis is full at 10 drives, about 1.35 drives worth of capacity is 'lost' to parity overhead/efficiency loss⁴.

That is quite a lot of overhead or loss of capacity, I think.

How is this overhead calculated? If we would just buy 10 drives and create a 10-drive RAIDZ2 vdev, data-to-parity overhead is 20% meaning that 20% of the total raw capacity of the vdev is used for storing parity. This is the most efficient scenario in this case.

When we start out with the four-drive RAIDZ2 vdev, the data-to-parity overhead is 50%. That's a 30% overhead difference compared to the 'ideal' 10-drive setup.

As we keep adding drives, the relative overhead of the parity keeps dropping so we end up with 'multiple data sets' with different data-to-parity ratios, that are less efficient than the end-stage of 10 drives.

I created a google sheet to roughly estimate this overhead for each stage, but my math was totally off. Fortunately, Yorick rewrote the sheet, which can be found here. Thanks Yorick! Further more, Truenas user DayBlur shared additional insights on the calculations if you are interested in that.

The google sheet allows you to play with various variables to estimate how much capacity is lost for a given scenario. Please note that any losses that may arise because a number of drives is used that requires data to be padded - as discussed in the Ars Technica article - are not part of the calculation.

It is a bit unfortunate that especially in the scenario of the home user who want to start small and expand-as-you go that this overhead manifests itself so much. But there is good news!

Lost capacity can be recovered!

The overhead or 'lost capacity' can be recovered by rewriting existing data after the vdev has been expanded, because the data will then be written with the more efficient parity-to-data ratio of the larger vdev.

Rewriting all data may take quite some time and you may opt to postpone this step until the vdev has been expanded a couple of times so the parity-to-data ratio is now 'good enough' that significant storage gains can be had by rewriting the data.

Because capacity lost to overhead can be fully recovered, I think that this caveat is relatively minor, especially compared to the old situation where we had to expand a pool with entire vdevs and there was no way to recover any overhead.

There is currently no build-in mechanism to trigger this data rewrite as part of the native ZFS tools. This will be a manual process until somebody may create a script that automates this process. According to Matthew Ahrens, restriping the data as part of the vdev expansion process would be an effort of similar scale as the RAIDZ expansion itself.

Evaluation

I think it cannot be stated enough how awesome the RAIDZ vdev expansion feature is, especially for home users who want to start small and grow their storage over time.

Although the expansion process can accumulate quite a bit of overhead, that overhead can be recovered by rewriting existing data, which is probably not a problem for most people.

Despite all the awesome features and capabilities of ZFS, I think quite a few home users went with other storage solutions because of the relatively high expansion cost/overhead. Now that this barrier will be overcome, I think that ZFS will be more accessible to the home user DIY NAS crowd.

Release timeline

According to the Ars Technica article by Jim Salter, this feature will probably become available in August 2022, so we need to have some patience. Even so, you might want to already decide to build your new DIY NAS based on ZFS: by the time you may need to expand your storage, the feature may be available!

Update on some - in my opinion - bad advice

The podcast 2.5 admins (which I enjoy listening to) discussed the topic of RAIDZ expansion in episode 45.

There are two remarks made that I want to address, because I disagree with them.

Don't rewrite the data?

As in his Ars Technica article, Jim Salter keeps advocating not to bother rewriting the data after a vdev expansion, but I personally disagree with this advice. I hope I have demonstrated that if you keep adding drives, the parity overhead is significant enough for most home users to make it worthwhile to rewrite the data after a few drives have been added.

Just use mirrors!

I also disagree with the advice of using mirrors, especially for home users⁶. I personally think it is bad advice, because home users have other needs and desires as enterprise environments.

If 'just use mirrors' is still the advice, why did Matthew Ahrends build the whole RAIDZ vdev expansion feature in the first place? I think the RAIDZ vdev expansion is really beneficial for home users.

Maybe Jim and I have very different ideas about what a home user would want or need in a DIY NAS storage solution. I think that home users want this:

As much storage as possible for as little money as possible with acceptable redundancy.

In addition, I think that home users in general work with larger files (multiple megabytes at least). And if they sometimes work with smaller files, they accept some performance loss due to the lower random I/O performance of single RAIDZ vdevs⁷.

Frankly, to me it feels like the 'just use mirrors' advice is used to 'downplay' a significant limitation of ZFS⁸. Jim is a prolific writer on Ars Technica and has a large audience so his advice matters. So that's why I think it's sad that he sticks with 'just use mirrors' while that's clearly not in the best interest of most home users.

However, that's just my opinion, you decide for yourself what's best.

The other method is to replace all existing drives one by one with larger ones. Only after you have replaced all drives will you be able to gain extra capacity so this method has a similar downside as just expanding with extra vdevs: you must buy multiple drives at once. In addition, I think this method is rather time consuming and cumbersome although people do use it to expand capacity. And to be fair: you can indeed add 4+ disk vdevs, vdevs with a higher RAIDZ level or mirrors but none of that makes sense in this context. ↩
Just to illustrate the level of redundancy in terms of how many disks can be lost and still be operational. ↩
I personally think that it's even great for small and medium business owners. Only larger businesses want to keep adding relatively large vdevs consisting of multiple drives because if they keep expanding with just one drive at a time, they may have to expand capacity very frequently which may not be practical. ↩
If you would only upgrade once the pool is almost full - not recommended! - that overhead grows to 1.69 drives. ↩
So you go from four to five drives. Then from five to six drives, and so on. ↩
I link to the original article by Jim Salter because I want to allow you to read the article and make up your own mind and not just listen to me. ↩
If random I/O performance is important, it is probably wise to go for SSD based storage anyway. ↩
resolved by by ZFS vdev expansion obviously, when it lands in production. ↩

Recycle your old laptop display and turn it into a monitor

2021-03-13T12:00:00+01:00

During a cleaning spree I decided that it was time to recycle two old laptops that were just collecting dust on a shelf for many years. Although I didn't have any purpose for them anymore, I realised that the displays were still perfectly fine.

This is the display of my old 13" Intel-based MacBook.

Somehow it felt wasteful to just throw the laptops out, I wondered if it would be possible to use these displays as a regular monitor?

It turns out: yes this is possible and it is also quite simple. My blogpost covers the same topic, with a few more pictures¹.

There is a particular LCD driver board that can be found all over ebay and the well-known Chinese webshops.

These boards cost around $20 and come with most things you need to get the display operational.

The board includes:

A small controller print for the on-screen display (center)
A high-voltage power supply for the backlight (left)
The data cable to drive the actual display (right)
Support for audio-passthrough + volume control (right corner)

The board doesn't include:

A 12-volt power supply
Some kind of frame to make the display and board one easy to handle unit

The board supports HDMI, VGA and DVI for signal input so it should work with almost any computer.

This particular board (M.NT68676.2) is used to power many different panel models. Although the board itself may be the same, it's important to order the board that is specifically tailored to the particular LCD panel you have. The panels seem easy to identify. This is the Macbook LCD panel type (LP133WX1 TL A1):

That TL/A1 part of the model is critical to finding the appropriate controller.

I also have an old Dell Vostro screen that uses the exact same driver board, but the cables are different. Also, it may be the case that the boards are flashed with the appropriate firmware for the particular panel. So I would recommend not to gamble and get the driver board that exactly matches the model number.

To make everything work, we first connect the high-voltage module to the backlight power cable...

...and we also connect the LCD driver cable:

When I connected everything for the first time, the display didn't work at all. It turns out that the board shipped with a second, separate interface cable and I had to swap those cables to make the display work properly.

Power supply

According to the sales page, the board requires a 12-volt 2A adapter, but in practice, I could get away with a tiny, much weaker power supply.

I found an old adapter from a Wi-Fi access-point (long gone) which is rated for just 0.5A. It did power both the board and the screen perfectly. It worked with both the Macbook display as the Dell display so it doesn't seem to be a fluke.

Although I didn't measure actual power consumption, we know that it can't be more than just 6 Watt because that's what the power adapter can handle.

Laptop displays need to be power efficient and that may also make them interesting for low-power or battery-powered projects.

We've only just begun

The display works, but that was the easy part.

The hardest part of this proces (not pictured) was switching the on-screen-display from Chinese (which I don't master) to English. But there is more work ahead.

At this point we end up with just a fragile LCD panel connected to a driver board through a bunch of wires. The whole setup is just an unpractical mess. There are at least two things left to do:

Mount the driver board, OSD controller and high-voltage unit to the back of the LCD panel
Make some kind of stand

For the old Dell display, I used a bit of wood and hot glue to make a wooden scaffold on which I could mount the driver board with a few screws. It won't win any prizes for sure but it's just an example of what needs to be done make the display (more) manageable.

It still doesn't have a stand but that's for another day. I can imagine that if you own a 3D printer, you can make a nice case with a stand, although that will increase the cost of the project.

Evaluation

What I like most about this kind of project is the fact that for very little money, you can recycle a perfectly fine and usable display that will probably last for another five to ten years. The project takes very little effort and it is also not difficult to do.

You can augment existing hobby projects with a screen and due to the relatively low power consumption, it may even be suitable for battery-powered projects.

And with a bit of work you can make a nice (secondary) monitor out of them. Finally you have an excuse to dust off one of your unused Raspberry Pis that you had to have but didn't have any actual use for.

The thrift-store is cheaper

If your goal is just to get a cheap LCD display, it may be cheaper to go to the nearest thrift-store and buy some old second-hand display for $10. But that may have some drawbacks:

It will be much larger than the laptop screen
It is powered by 110/220 volt so less suitable for a battery-powered setup
overall power consumption will be higher

So it all depends on your particular needs.

Closing words

If you also repurposed a laptop monitor for a project or just as a (secondary) screen, feel free to share your work in the comments.

I posted the article on slashdigit to hacker news and it got quite an interest. ↩

Understanding the Ubuntu 20.04 LTS Server Autoinstaller

2021-02-11T12:00:00+01:00

Introduction

Ubuntu Server version 18.04 LTS uses the debian-installer (d-i) for the installation process. This includes support for 'preseeding' to create unattended (automated) installations of ubuntu.

the debian installer

With the introduction of Ubuntu Server 20.04 'Focal Fossa' LTS, back in April 2020, Canonical decided that the new 'subiquity server installer' was ready to take it's place.

After the new installer gained support for unattended installation, it was considered ready for release. The unattended installer feature is called 'Autoinstallation'.

I mostly run Ubuntu 18.04 LTS installations but I decided in February 2021 that I should get more acquainted with 20.04 LTS, especially when I discovered that preseeding would no longer work.

In this article I assume that the reader is familiar with PXE-based unattended installations.

20.04 LTS and 22.04 LTS Can't install on USB drive (Update Aug 2022)

I run a few HP Microservers that boot from a SATA SSD in an USB drive enclosure using the internal USB header. Automated installs for Ubuntu 18.04 LTS have no problem installing and booting from a USB device.

Unfortunately, both Ubuntu 20.04 and 22.04 LTS install fine, but no matter what I do, they won't boot from USB. I've tried different enclosures and most of then won't work. Only one powered USB dock (that is way to big to fit inside the Microserver) does work and let 22.04 boot from USB.

Again: Ubuntu 18.04 and the Latest Debian work fine so this seems an issue specific to the new Autoinstall mechanism.

Note: I didn't try this on other hardware (which I don't have) so it might be an issue specific to the HP Microserver Gen8.

Why this new installer?

Canonical's desire to unify the codebase for Ubuntu Desktop and Server installations seems to be the main driver for this change.

From my personal perspective, there aren't any new features that benefit my use-cases, but that could be different for others. It's not a ding on the new Autoinstaller, it's just how I look at it.

There is one conceptual difference between the new installer and preseeding. A preseed file must answer all questions that the installer needs answered. It will switch to interactive mode if a question is not answered, breaking the unattended installation process. From my experience, there is a bit of trial-and-error getting the preseed configuration right.

The new Subiquity installer users defaults for all installation steps. This means that you can fully automate the installation proces with just a few lines of YAML. You do't need an answer for each step.

The new installer as other features such as the ability to SSH into an installer session. It works by generating an at-hoc random password on the screen / console which you can use to logon remotely over SSH. I have not used it yet as I never found it necessary.

Documentation is a bit fragmented

As I was trying to learn more about the new Autoinstaller, I noticed that there isn't a central location with all relevant (links to) documentation. It took a bit of searching and following links, to collect a set of useful information sources, which I share below.

Link	Description
Reference Manual	Reference for each particular option of the user-data YAML
Introduction	An overview of the new installer with examples
Autoinstall quick start	Example of booting a VM using the installer using KVM
Netbooting the installer	A brief instruction on how to setup PXE + TFTP with dnsmasq in order to PXE boot the new installer
Call for testing	Topic in which people give feedback on their experience with the installer (still active as of February 2021) with a lot of responses
Stack Exchange post	Detailed tutorial with some experiences.
Medium Article	Contains an example and some experiences.
github examples	Github repo with about twenty examples of more complex configurations

The reference documentation only supports some default use-cases for the unattended installation process. You won't be able to build more complex configuration such as RAID-based installations using this reference.

Under-the-hood, the installer uses curtin. The linked documentation can help you further build complex installations, such as those which use RAID.

I think the curtin syntax is a bit tedious and fortunately it is probably not required to learn curtis and piece together more complex configurations by hand. There is a nice quality-of-life feature that takes care of this.

How does the new installer work?

With a regular PXE-based installation, we use the 'netboot' installer, which consists of a Linux kernel and an initrd image (containing the actual installer).

This package is about 64 Megabytes for Ubuntu 18.04 LTS and it is all you need, assuming that you have already setup a DHCP + TFTP + HTTP environment for a PXE-based installation.

The new Subiquity installer for Ubuntu 20.04 LTS deprecates this 'netboot' installer. It is not provided anymore. Instead, you have to boot a 'live installer' ISO file which is about 1.1 GB in size.

The process looks like this:

Download the Live installer ISO
Mount the iso to acquire the 'vmlinuz' and 'initrd' files for the TFTP root
Update your PXE menu (if any) with a stanza like this:

LABEL focal2004-preseed
                MENU LABEL Focal 20.04 LTS x64 Manual Install
                KERNEL linux/ubuntu/focal/vmlinuz
                INITRD linux/ubuntu/focal/initrd
                APPEND root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=http://10.10.11.1/ubuntu-20.04.1-live-server-amd64.iso

This process is documented here with detailed how-to steps and commands.

We have not yet discussed the actual automation part, but first we must address a caveat.

The new installer requires 3GB of RAM when PXE-booting

UPDATE May 2022:

The 20.04.4 update won't work on a machine with 4096 MB of memory when PXE booting. My testing shows that with 20.04 update 04 requires at least 4300MB of memory.

So if we talk about a physical machine, based on regular memory DIMM sizes, it's likely that the machine must have 6 to 8 GB of memory. Just to install Ubuntu Linux over PXE. I think that's not right.

Although it is not explicitly documented¹, the new mechanism of PXE-booting the new Ubuntu installer using the Live ISO requires a minimum of 3072 MB of memory. And I assure you, 3000 MB is not enough.

It seems that the ISO file is copied into memory over the network and extracted on the RAM disk. With a RAM disk of 1500 MB and an ISO file of 1100 MB we are left with maybe 472 MB of RAM for the running kernel and initrd image.

To put this into perspective: I could perform an unattended install of Ubuntu 18.04 LTS with only 512 MB of RAM².

Due to this 'new' installation process, Ubuntu Server 20.04 has much more demanding minimum system requirements than Windows 2019 Server, which is fine with 'only' 512 MB of RAM, even during installation. I have to admit I find this observation a bit funny.

It seems that this 3 GB memory requirement for the installation process is purely and solely because of the new installation process. Obviously Ubuntu 20.04 can run in a smaller memory footprint once installed.

Under the hood, a tool called 'casper' is used to bootstrap the installation process and that tool only supports a local file system (in this case on a RAM disk). On paper, casper does support installations using NFS or CIFS but that is not supported nor tested. From what I read, some people tried to use it but it didn't work out.

As I undertand it, the curent status is that you can't install Ubuntu Server 20.04 LTS on any hardware with less than 3GB of memory using PXE-boot. This probably affects older and less potent hardware, but I can remember a time that this was actually part of the point of running Linux.

It just feels wrong conceptually, that a PXE-based server installer requires 3GB of memory.

Important

The 3GB memory requirement is only valid for PXE-based installations. If you boot from ISO / USB stick you can install Ubuntu Server on a system with less memory. I've verified this with a system with only 1 GB of memory.

The Autoinstall configuration

Now let's go back to the actual automation part of Autoinstaller. If we would want to automate our installation, our PXE menu item should be expanded like this:

APPEND root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=http://10.10.11.1/ubuntu-20.04.1-live-server-amd64.iso autoinstall ds=nocloud-net;s=http://10.10.11.1/preseed/cloud-init/

In this case, the cloud-init folder - as exposed through an HTTP-server - must contain two files:

meta-data
user-data

The meta-data file contains just one line:

instance-id: focal-autoinstall

The user-data file is the equivalent of a preseed file, but it is YAML-based instead of just regular plain text.

Minimum working configuration

According to the documentation, this is a minimum viable configuration for the user-data file:

#cloud-config
autoinstall:
  version: 1
  identity:
    hostname: ubuntu-server
    password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
    username: ubuntu

This works fine and performs a basic install + apt upgrade of the new system with a disk configuration based on an LVM layout.

My preferred minimum configuration:

Personally, I like to keep the unattended installation as simple as possible. I use Ansible to do the actual system configuration, so the unattended installation process only has to setup a minimum viable configuration.

I like to configure the following parameters:

Inject a SSH public key into the authorized_keys file for Ansible
Configure the Apt settings to specify which repository to use during installation (I run a local debian/ubuntu mirror)
Update to the latest packages during installation

I'll give examples of the required YAML to achieve this with the new installer.

Injecting a public SSH key for the default user:

ssh:
    authorized-keys: |
       ssh-rsa <PUB KEY>
    install-server: true
    allow-pw: no

Notice that we also disable password-authentication for SSH access.

Configure APT during installation

I used this configuration to specify a particular mirror for both the installation process and for the system itself post-installation.

Mirror:
  mirror: "http://mirror.mynetwork.loc"
apt:
  preserve_sources_list: false
  primary:
    - arches: [amd64]
      uri: "http://mirror.mynetwork.loc/ubuntu"

Performing an apt upgrade

By default, the installer seems to install security updates but it doesn't install the latest version of softare. This is a deviation from the d-i installer which always ends up with a fully up-to-date system, when done.

The same end-result can be accomplished by running an apt update and apt upgrade at the end of the installation process.

late-commands:
- curtin in-target --target=/target -- apt update           
- curtin in-target --target=/target -- apt upgrade -y

So all in all, this is not a big deal.

Network configuration

The network section can be configured using regular Netplan syntax. An example:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: no
      addresses:
        - 10.10.50.200/24
      gateway4: 10.10.50.1
      nameservers:
        search:
          - mynetwork.loc
        addresses:
          - 10.10.50.53
          - 10.10.51.53

Storage configuration

The installer only supports a 'direct' or 'lvm' layout. It also selects the largest drive in the system as the boot drive, for installation.

If you want to setup anything more complex as you may want to setup RAID or a specific partion layout, you need to use the curtin syntax.

It is not immediately clear how to setup a RAID configuration based on the available documentation.

Fortunately, the new installer supports creating a RAID configuration or custom partition layout if you perform a manual install.

It turns out that when the manual installation is done you can find the cloud-init user-data YAML for this particular configuration in the following file:

    /var/log/installer/autoinstall-user-data

I think this is extremely convenient.

So to build a proper RAID1-based installation, I followed these instructions.

So what does the YAML for this RAID setup look like?

This is the storage section of my user-data file (brace yourself):

storage:
    config:
    - {ptable: gpt, serial: VBOX_HARDDISK_VB50546281-4e4a6c24, path: /dev/sda, preserve: false,
      name: '', grub_device: true, type: disk, id: disk-sda}
    - {ptable: gpt, serial: VBOX_HARDDISK_VB84e5a275-89a2a956, path: /dev/sdb, preserve: false,
      name: '', grub_device: true, type: disk, id: disk-sdb}
    - {device: disk-sda, size: 1048576, flag: bios_grub, number: 1, preserve: false,
      grub_device: false, type: partition, id: partition-0}
    - {device: disk-sdb, size: 1048576, flag: bios_grub, number: 1, preserve: false,
      grub_device: false, type: partition, id: partition-1}
    - {device: disk-sda, size: 524288000, wipe: superblock, flag: '', number: 2, preserve: false,
      grub_device: false, type: partition, id: partition-2}
    - {device: disk-sdb, size: 524288000, wipe: superblock, flag: '', number: 2, preserve: false,
      grub_device: false, type: partition, id: partition-3}
    - {device: disk-sda, size: 1073741824, wipe: superblock, flag: '', number: 3,
      preserve: false, grub_device: false, type: partition, id: partition-4}
    - {device: disk-sdb, size: 1073741824, wipe: superblock, flag: '', number: 3,
      preserve: false, grub_device: false, type: partition, id: partition-5}
    - {device: disk-sda, size: 9136242688, wipe: superblock, flag: '', number: 4,
      preserve: false, grub_device: false, type: partition, id: partition-6}
    - {device: disk-sdb, size: 9136242688, wipe: superblock, flag: '', number: 4,
      preserve: false, grub_device: false, type: partition, id: partition-7}
    - name: md0
      raidlevel: raid1
      devices: [partition-2, partition-3]
      spare_devices: []
      preserve: false
      type: raid
      id: raid-0
    - name: md1
      raidlevel: raid1
      devices: [partition-4, partition-5]
      spare_devices: []
      preserve: false
      type: raid
      id: raid-1
    - name: md2
      raidlevel: raid1
      devices: [partition-6, partition-7]
      spare_devices: []
      preserve: false
      type: raid
      id: raid-2
    - {fstype: ext4, volume: raid-0, preserve: false, type: format, id: format-0}
    - {fstype: swap, volume: raid-1, preserve: false, type: format, id: format-1}
    - {device: format-1, path: '', type: mount, id: mount-1}
    - {fstype: ext4, volume: raid-2, preserve: false, type: format, id: format-2}
    - {device: format-2, path: /, type: mount, id: mount-2}
    - {device: format-0, path: /boot, type: mount, id: mount-0}

That's quite a long list of instructions to 'just' setup a RAID1. Although I understand all the steps involved, I'd never have come up with this by myself on short notice using just the documentation, so I think that the autoinstall-user-data file is a life saver.

After the manual installation, creating a RAID1 mirror, I copied the configuration above into my own custom user-data YAML. Then I performed an unattended installation and it worked on the first try.

So if you want to add LVM into the mix or make some other complex storage configuration, the easiest way to automate it, is to first do a manual install and then copy the relevant storage section from the autoinstall-user-data file to your custom user-data file.

Example user-data file for download

I've published a working user-data file that creates a RAID1 here.

You can try it out by creating a virtual machine with two (virtual) hard drives. I'm assuming that you have a PXE boot environment setup.

Obviously, you'll have to change the network settings for it to work.

Generating user passwors

If you do want to logon onto the console with the default user, you must generate a salt+password hash and copy/past that into the user-data file.

I juse the 'mkpasswd' command for this like so:

mkpasswd -m sha-512

The mkpasswd utility is part of the 'whois' package.

Closing words

For those who are still on Ubuntu Server 18.04 LTS, there is no need for immediate action as this version is supported until 2023. Only support for new hardware has come to an end for the 18.04 LTS release.

At some point, some time and effort will be required to migrate towards the new Autoinstall solution. Maybe this blogpost helps you with this transition.

It took me a few evenings to master the new user-data solution, but the fact that a manual installation basically results in a perfect pre-baked user-data file³ is a tremendous help.

I think I just miss the point of all this effort of revamping installers but maybe I'm not hurt by the limitations of the older existing solution. If you have any thoughts on this, feel free to let me know in the comments.

I had to discover this information on StackExchange somewhere down below in this very good article after experiencing problems. ↩
I tried 384 MB and it didn't finish, just got stuck. ↩
/var/log/installer/autoinstall-user-data ↩

A 12.48 inch (1304x984) three-color e-paper display by Waveshare

2020-12-22T12:00:00+01:00

Introduction

Update September 2023: A small horizontal and large vertical row of pixels died. Demo of defective display here.

I'm running a solar-powered blog and I wanted to add a low-power display to show the daily solar 'harvest'¹ and maybe some additional information.

So I decided to use an e-paper display. I wanted a display that would be readable from a distance, so bigger would be better. I therefore chose the Waveshare 12.48 inch e-paper display².

an example based on data from the summer, generated with Graphite

This particular display costs $179 excluding taxes and shipping at the time this article was written.

Specifications

Waveshare sells a two-color (black and white) and a three-color version (black, white, red). I bought the three-color version. The three-color version is the (B) model.

Specifications:

Screen size     :      12.48 inches
Resolution      :      1304 x 984
Colors          :      black, white, red
Greyscale       :      2 levels
Refresh rate    :      16 seconds
Partial refresh :      Not supported
Interfaces      :      Raspberry Pi, ESP32, STM32, Arduino

The two-color variant of this display has a refresh rate of 8 seconds.

This display is clearly quite slow. Furthermore, the lack of partial refresh support could make this display unsuitable for some applications. I was OK with this slow refresh rate.

The image below demonstrates different fonts and sizes. I think DejaVuSansMono-Bold looks really well on the display, better than the font supplied by Waveshare.

The Interfaces

The display includes a microcontroller that in turn can be driven through one of four interfaces:

A Raspberry Pi (Worked)
An ESP32 (Not tested)
An Arduino (Didn't work)
An STM32 (Not tested)

I've tried the Arduino header with an Arduino Uno, but the supplied demo code didn't work. I did not investigate further why this was the case. It could be a problem with voltage regulation.

In the image above, the black plastic backplate is removed.

Image quality

These e-paper displays are mostly sold as product information displays for supermarkets and other businesses. However, the quality is good enough to display images. Especially the support for red can make an image stand out.

Below is an example of an image that incorporates the third (red) color.

The display seems suitable to display art.

It looks quite good in real life (sorry for the glare).

How the display shows three colors

The display acts like it is actually two displays in one. A black and white display, and a red and white display.

First, the black and white image is drawn. Next, the red and white image is put on top.

Because the display has to draw two images in succession, it takes 16 seconds to refresh the screen. This explains why the black-and-white version of this screen does a refresh in eight seconds: it doesn't have to refresh the red color.

Please note that the entire process of displaying content on the screen takes much longer.

A demonstration:

Displaying an image is cumbersome (On Raspberry Pi 3B+)

At the time this article was written, I could not find any information or tools for this display³.

Many Waveshare e-paper displays are popular and have a decent community support. However, it seems that this display is rather unknown.

Therefore, it seems that there are no tools available to display an arbitrary image on this display. You can use the example Python code to display an image but you have to follow these steps:

Create a black-and-white version of the image
Create a red-and-white version of the image, that contains only data for the red parts of the image
If the source image doesn't match the required resolution, you have to resize, crop and fill the image where appropriate.

Both 'black' and 'red' images need to exactly match the resolution of the display (1304x984) or the library will abort with an error.

As I found this proces tedious, I automated it.

A new tool to make displaying an image easy

I've used the python library as supplied by Waveshare and created a command-line tool (Github) on top of it to perform all the required steps as described in the previous section. I'm using Imagemagick for all the image processing.

The script works like this:

./display -i <image file> [--rotate 90] [--fuzz 35] [--color yellow]

The --fuzz and --color parameters may require some clarification.

The color red is extracted from an image but it's not always perfect. By applying the --fuzz parameter (the argument is a percentage), it is possible to capture more of the red (or selected color) of an image.

The --color option specifies which color should be 'converted' to red. By default this color is 'red' (obviously). The 'solar chart' (at the start of this article) is an example where a yellow line was converted to red.

Very slow: it takes about 55 seconds to display an image using the Raspberry Pi 3B+. Half of that minute is spend converting the images to the appropriate format using Imagemagick.

Informational: The Python program includes a modified version of the Waveshare Python library. This library has been altered to prevent double conversion of images, which significantly degrades image quality.

Slow performance

If you use the provided Python library (Python3 compatible) it takes about 30+ seconds to draw an image on the screen. (This excludes the image processing performed with the 'display' tool.)

Further testing showed that the Python library converts and dithers the image before it is sent to the display. And it does so for both black and red. Dithering is performed by looping in Python over every of the 1.3 milion pixels.

Each of these loops (for black and red) take about 10 seconds on the Raspberry Pi 3B+, which explains why it takes so long to update the display. Therefore, I think the combination of Python + the Raspberry Pi 3B+ is not ideal in this case.

Evaluation

I wanted to share my experience with this display to make other people aware of its existence. The tool I created should make it simple to get up and running and display an image.

It clearly has some drawbacks but due to the size, resolution and third color, it seems to be unique and may therefore be interesting.

Although I never tried the display with an ESP32, I think its an ideal for the purpose of a low-power picture frame.

This article was discussed on Hacker News (briefly). This resulted in about 9000 unique visitors for this article.

Appendix A - Remark on other displays

Please note: Waveshare also sells a smaller 10.3 inch black-and-white e-paper display for a similar price with some significant benefits:

Screen size     :      10.3 inches
Resolution      :      1872 x 1404
Colors          :      black and white
Greyscale       :      16 levels
Refresh rate    :      450 miliseconds
Partial refresh :      Supported

This particular display is smaller but has a higher resolution, supports 16 grayscale levels and updates in half a second. This display may better suit your particular needs. For example, I believe that this display may have been used in this project, a solar-powered digital photo frame.

Appendix B - How to make the display work on a Raspberry Pi

This information is straight from the Waveshare site but I include it for completeness and ease of use.

PART I: Enable the SPI interface with raspi-config

sudo raspi-config
select Interfacing Options
Select SPI
Select Yes
Reboot the Raspberry Pi

PART II: Install the required libraries

Install BCM283

Website

wget http://www.airspayce.com/mikem/bcm2835/bcm2835-1.60.tar.gz
tar zxvf bcm2835-1.60.tar.gz 
cd bcm2835-1.60/
sudo ./configure
sudo make
sudo make check
sudo make install

Install wiringPi

website

sudo apt-get install wiringpi
cd /tmp
wget https://project-downloads.drogon.net/wiringpi-latest.deb
sudo dpkg -i wiringpi-latest.deb
gpio -v

Caution: The library seems to be deprecated. The Raspberry Pi 4 is supported but future versions of the Raspberry Pi may not.

The wiringPi library is used as part of a compiled library called "DEV_Config.so" as found in the ./lib directory.

@raspberrypi:~/epaper_display $ ldd lib/DEV_Config.so 
    linux-vdso.so.1 (0x7ee0d000)
    /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f1e000)
    libwiringPi.so => /usr/lib/libwiringPi.so (0x76f00000)
    libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76e7e000)
    libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76d30000)
    libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76d06000)
    librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76cef000)
    libcrypt.so.1 => /lib/arm-linux-gnueabihf/libcrypt.so.1 (0x76caf000)
    /lib/ld-linux-armhf.so.3 (0x76f46000)

Install Python3 and required libraries

sudo apt-get update
sudo apt-get install python3-pip
sudo apt-get install python3-pil
sudo pip3 install RPi.GPIO
sudo pip3 install spidev

Apendix C - E-paper hacking

I see myself as a consumer and I don't have any desire to hack the display for lower refresh-rates or partial refresh support, with the risk of damaging the display in the process.

However one resource about this topic I find very informative is a video from the Youtube Channel "Applied Science" (By Ben Krasnow), called "E-paper hacking: fastest possible refresh rate".

Apendix D - Available libraries

Example code for all supported platforms can be found in this github location.

I also found this github repository that may support this display. This code (also) didn't work for me on my Arduino uno. This could be due to a voltage mismatch, but I'm not willing to solder and potentially destroy the display.

Apendix E - Links to other e-paper projects

Very large and expensive display (Medium paywall)

E-paper calendar

Solar-powered digital photo frame

During fall and winter, there is almost no power generation due to the very sub-optimal location of my balcony. ↩
You can go larger but at a cost. ↩
My google skills may be at fault but that point is now moot. ↩

Most Technical debt is just bullshit

2020-09-25T12:00:00+02:00

Introduction

I made an offhand remark about technical debt to a friend and he interrupted me, saying: "technical debt is just bullshit". In his experience, people talking about technical debt were mostly trying to:

cover up bad code
cover up unfinished work

source¹

Calling these issues 'technical debt' seems to be a tactic of distancing oneself from these problems. A nice way of avoiding responsibility. To sweep things under the rug.

Intrigued, I decided to take a better look at the metaphor of techical debt, to better understand what is actually meant.

Tip: this article on Medium by David Vandegrift also tackles this topic.

A definition of technical debt

Right off the bat, I realised that my own understanding of technical debt was wrong. Most people seem to understand technical debt as:

"cut a corner now, to capture short-term business value (taking on debt), and clean up later (repaying the debt)".

I think that's wrong.

Ward Cunningham, who coined the metaphor of technical debt, wrote:

You know, if you want to be able to go into debt that way by developing software that you don't completely understand, you are wise to make that software reflect your understanding as best as you can, so that when it does come time to refactor, it's clear what you were thinking when you wrote it, making it easier to refactor it into what your current thinking is now.

In some sense, this reads to me as a form of prototyping. To try out and test design/architecture to see if it fits the problem space at hand. But it also incorporates the willingness to spend extra time in the future to change the code to better reflect the current understanding of the problem at hand.

... if we failed to make our program align with what we then understood to be the proper way to think about our financial objects, then we were gonna continually stumble over that disagreement and that would slow us down which was like paying interest on a loan.

The misalignment of the design/architecture and the problem domain creates a bottleneck, slowing down future development.

So I think it's clearly not about taking shortcuts for a short-term business gain.

It is more a constant reinvestment in the future. It may temporary halt feature work, but it should result in more functionality and features in the long run. It doesn't seem short-term focussed at all to me. And you need to write 'clean' code and do your best because it is likely that you will have to rewrite parts of it.

These two articles by Ron Jeffries already discuss this in great detail.

A logical error

Reading up on the topic, I noticed something very peculiar. Somehow along the way, everything that hinders software development has become 'technical debt'.

Anything that creates a bottleneck, is suddenly put into the basket of technical debt. I started to get a strong sense that a lot of people are somehow making a logical fallacy.

If you have technical debt, you'll experience friction when trying to ignore it and just plow ahead. The technical debt creates a bottleneck.

But then people reason the wrong way around: I notice a bottleneck in my software development process, so we have 'technical debt'.

However, because technical debt creates a bottleneck, it doesn't follow that every bottleneck is thus technical debt.

I think it's this flawed reasoning that turns every perceived obstacle into technical debt².

Maybe I'm creating a straw man argument, but I think I have some examples that show that people are thinking the wrong way around.

If we look at the wikipedia page about technical debt, there is a long list of possible causes of technical debt.

To site some examples:

Insufficient up-front definition
Lack of clear requirements before the start of development
Lack of documentation
Lack of a test suite
Lack of collaboration / knowledge sharing
Lack of knowledge/skills resulting in bad or suboptimal code
Poor technical leadership
Last minute specification changes

Notice that these issues are called 'technical debt' because they can have a similar outcome as technical debt. They can create a bottleneck.

But why the hell would we call these issues technical debt?

These issues are self-explanatory. Calling them technical debt not only seems inappropriate, it just obfuscates the cause of these problems and it doesn't provide any new insight. Even in conversations with laypeople.

A mess is not a Technical Debt

A blogpost by Uncle Bob with the same title³ also hits on this issue that a lot of issues are incorrectly labeled as 'technical debt'.

Unfortunately there is another situation that is sometimes called “technical debt” but that is neither reasoned nor wise. A mess.

...

A mess is not a technical debt. A mess is just a mess. Technical debt decisions are made based on real project constraints. They are risky, but they can be beneficial. The decision to make a mess is never rational, is always based on laziness and unprofessionalism, and has no chance of paying of in the future. A mess is always a loss.

Cunningham's definition of technical debt shows that it's a very conscious and deliberate process. Creating a mess isn't. It's totally inappropriate to call that technical debt. It's just a mess.

I think that nicely relates back to that earlier list from wikipedia. Just call things out for what they actually are.

Is quibbling over 'technical debt' as a metaphor missing the point?

In this blogpost, Martin Fowler addresses the blogpost by Uncle Bob and argues that technical debt as a metaphor is (still) very valuable when communicating with non-technical people.

He even introduces a quadrant:

	Reckless	Prudent
Deliberate	"We don't have time for design"	"We must ship now and deal with consequences (later)"
inadvertent	"What's Layering?"	"Now we know how we should have done it"

This quadrant makes me extremely suspicious. Because in this quadrant, everything is technical debt. He just invents different flavours of technical debt. It's never not technical debt. It's technical debt all the way down.

It seems to me that Martin Fowler twists the metaphor of technical debt into something that can never be falsified, like psychoanalysis.

It's not 'bad code', a 'design flaw' or 'a mess', it's 'inadvertent & reckless technical debt'. What is really more descriptive of the problem?

Maybe it's just my lack of understanding, but I fail to see why it is in any way helpful to call every kind of bottleneck 'technical debt'. I again fail to see how this conveys any meaning.

In the end, what Fowler does is just pointing out that bottlenecks in software development can be due to the four stages of competence.

	Incompetence	Competence
Concious	"We don't have time for design"	"We must ship now and deal with consequences (later)"
Unconscious	"What's Layering?"	"Now we know how we should have done it"

I don't think we need new metaphors for things we (even laypeople) already understand.

Does technical debt (even) exists?

The HFT Guy goes as far as to argue that technical debt doesn't really exists, it isn't a 'real' concept.

After decades of software engineering, I came to the professional conclusion that technical debt doesn’t exist.

His argument boils down to the idea that what people call technical debt is actually mostly maintenance.

So reincorporating a better understanding of the problem at hand into the code (design) is seen as an integral and natural part of software development, illustrated by the substitute metaphor of mining (alternating between digging and reinforcing). At least that's how I understand it.

Substituting one metaphor with another, how useful is that really? But in this case it's at least less generic and more precise.

Closing words

Although Cunningham meant well, I think the metaphor of technical debt started to take on a life of its own. To a point where code that doesn't conform to some Platonic ideal is called technical debt⁴.

Every mistake, every changing requirement, every tradeoff that becomes a bottleneck within the development process is labeled 'technical debt'. I don't think that this is constructive.

I think my friend was right: the concept of technical debt has become bullshit. It doesn't convey any better insight or meaning. On the contrary, it seems to obfuscate the true cause of a bottleneck.

At this point, when people talk about technical debt, I would be very sceptical and would want more details. Technical debt doesn't actually explain why we are where we are. It has become a hollow, hand-wavy 'explanation'.

With all due respect to Cunningham, because the concept is so widely misunderstood and abused, it may be better to retire it.

I discovered this image in this blogpost. ↩
if you are not working on a new feature, you are working on technical debt. ↩
I think that Uncle Bob's definition of technical debt in this article is not correct. He also defines it basically as cutting corners for short-term gain. ↩
See again Martin Fowlers article about technical debt. ↩

This blog is now running on solar power

2020-07-06T12:00:00+02:00

Introduction

This blog is now running on solar power.

I've put a solar panel on my balcony, which is connected to a solar charge controller. This device charges an old worn-out car battery and provides power to a Raspberry Pi ~~3b+~~ 4B, which in turn powers this (static) website.

For updates: scroll to the bottom of this article.

Some statistics about the current status of the solar setup is shown in the sidebar to the right. The historical graph below is updated every few minutes (European time).

Low-tech Magazine as inspiration

If you think you've seen a concept like this before, you are right.

The website Low-tech Magazine is the inspiration for my effort. I would really recommend visiting this site because it goes to incredible length to make the site energy-efficient. For example, images are dithered to save on bandwidth!

Low-tech Magazine goes off-line when there isn't enough sunlight and the battery runs out, which can happen after a few days of bad weather.

In January 2020, the site shared some numbers about the sustainability of the solar-powered website.

The build

My build is almost identical to that of Low-tech Magazine in concept, but not nearly as efficient. I've just performed a lift-and-shift of my blog from the cloud to a Raspberry Pi.

I've build my setup based on some parts I already owned, such as the old car battery and the Pi. The solar panel and solar charge controller were purchased new. The LCD display and current/voltage sensor have been recycled from an earlier hobby project.

I've used these parts:

Solar Panel	Monocrystalline 150 Watt 12V
Battery	12 Volt Lead Acid Battery (Exide 63Ah)
Solar Charge Controller	Victron BlueSolar MPPT 75\|10
Voltage/Current sensor	INA260
LCD Display	HD44780 20x4
Computer	Raspberry Pi 4B
Communications cable	VE.Direct to USB interface

The Solar Panel

The panel is extremely over-dimensioned because my balcony is directed towards the west, so it has only a few hours a day of direct sunlight. Furthermore, the angle of the solar panel is sub-optimal.

My main concern will be the winter. It is not unlikely that during the winter, the panel will not be able to generate enough energy to power the Pi and charge the battery for the night.

I have also noticed that under great sunlight conditions, the panel can easily produce 60+ Watt¹ but the battery cannot ingest power that fast.

I'm not sure about the actual brand of the panel, it was the cheapest panel I could find on Amazon for the rated wattage.

The Solar Charger

It's a standard solar charger made by Victron, for small solar setups (to power a shed or mobile home). I've bought the special data cable² so I can get information such as voltage, current and power usage.

The controller uses a documented protocol called ve.direct. I'm using a Python module to obtain the data.

According to the manual, this solar charger will assure that the battery is sufficiently charged and protects against deep discharge or other conditions that could damage the battery.

I feel that this is a very high-quality product. It seems sturdy and the communications port (which even supports a bluetooth dongle) giving you access to the data is really nice.

The controller is ever so slightly under-dimensioned for the solar panel, but since I will never get the theoretical full power of the panel due to the sub-optimal configuration, this should not be an issue.

The battery

In the day and age of Lithium-ion batteries it may be strange to use a Lead Acid battery. The fact is that this battery³ was free and - although too worn down for a car - can still power light loads for a very long time (days). And I could just hook up a few extra batteries to expand capacity (and increase solar energy absorption rates).

To protect against short-circuits, the battery is protected by a fuse. This is critical because car batteries can produce so much current that they can be used for welding. They are dangerous.

If you ever work with lead acid batteries, know this: don't discharge them beyond 50% of capacity, and ideally not beyond 70% of capacity. The deeper the discharge, the lower the life expectancy. A 100% discharge of a lead acid battery will kill it very quickly.

You may understand why Lead Acid batteries aren't that great for solar usage, because you need to buy enough of them to assure you never have to deep discharge them.

Voltage, Current and Power Sensor

I noticed that the load current sensor of the solar charge controller was not very precise, so I added an INA260 based sensor. This sensor uses I2C for communication, just like the LCD display. It measures voltage, current and power in a reasonable presice resolution.

Using the sensor is quite simple (pip3 install adafruit-circuitpython-ina260):

#!/usr/bin/env python3
import board
import adafruit_ina260
i2c = board.I2C()
ina260_L = adafruit_ina260.INA260(i2c,address=64)
print(ina260_L.current)
print(ina260_L.voltage)
print(ina260_L.power)

Please note that this sensor is purely optional, the precision it provides is not really required. I've used this sensor to observe that the voltage and current sensing sensors of the solar charge controller are fairly accurate, except for that of the load, which only measures the current in increments of 100 mAh.

The LCD Display

The display has four lines of twenty characters and uses a HD44780 controller. It's dirt-cheap and uses the I2C bus for communications. By default, the screen is very bright, but I've used a resistor on a header for the backlight to lower the brightness.

I'm using the Python RPLCD library (pip3 install RPLCD) for interfacing with the LCD display.

Using an LCD display in any kind of project is very simple.

#!/usr/bin/env python3
from RPLCD.i2c import CharLCD
lcd = CharLCD('PCF8574', 0x27, cols=20, rows=4)
lcd.clear()
lcd.cursor_pos = (0,0) # (line,column)
lcd.write_string("Hello")

12 volt to 5 Volt conversion

I'm just using a simple car cigarette lighter USB adapter to power the Raspberry Pi. I'm looking at a more power-efficient converter, although I'm not sure how much efficiency I'll be able to gain, if any.

Update: I've replaced the cigarette lighter usb adapter device with a buck converter, which resulted in a very slight reduction in power consumption.

Script to collect data

I've written a small Python script to collect all the data. The data is send to two places:

It is send to Graphite/Grafana for nice charts (serves no real purpose)
It is used to generate the infographic in the sidebar to the right

Because I don't want to wear out the SD card of the Raspberry Pi, the stats as shown in the sidebar to the right is written to a folder that is mounted on tmpfs.

The cloud as backup

When you connect to this site, you connect to a VPS running HAProxy. HAproxy determines if my blog is up and if so, will proxy between you and the Raspberry Pi. If the battery would run out, HAProxy will redirect you an instance of my blog on the same VPS (where it was running for years).

As you may understand, I still have to pay for the cloud VPS and that VPS also uses power. From an economical standpoint and from a ecological standpoint, this project may make little sense.

Possible improvements

VPS on-demand

The obvious flaw in my whole setup is the need for a cloud VPS that is hosting HAProxy and a backup instance of my blog.

A better solution would be to only spawn a cloud VPS on demand, when power is getting low. To move visitors to the VPS, the DNS records should be changed to point to the right IP-address, which could be done with a few API calls.

I could also follow the example of Low-tech Magazine and just accept that my blog would be offline for some time, but I don't like that.

Switching to Lithium-ion

As long as the car battery is still fine, I have no reason to switch to Lithium-ion. I've also purchased a few smaller Lead Acid batteries just to test their real-life capacity, to support projects like these. Once the car battery dies, I can use those to power this project.

The rest of the network is not solar-powered

The switches, router and modem that supply internet access are not solar-powered. Together, these devices use significantly more power, which I cannot support with my solar setup.

I would have to move to a different house to be able to install sufficient solar capacity.

Other applications

During good weather conditions, the solar panel provides way more power than is required to keep the battery charged and run the Raspberry Pi.

I've used the excess energy to charge my mobile devices. Although I think that's fun, if I just forget turning off my lights or amplifier for a few hours, I would already waste most of my solar gains.

I guess it's the tought that counts.

Conclusion

In the end, it it was a fun hobby project for me to realise. I want to thank Low-tech Magazine for the idea, I had a lot of fun creating my (significantly worse) copy of it.

If you have any ideas on how to improve this project, feel free to comment below or email me.

This blog post featured on hacker news and the Pi 3b+ had no problems handling the load.

Updates

Car battery died

After about two weeks the old and worn-down car battery finally died. Even after a whole day of charging, the voltage of the battery dropped to 11.5 Volts in about a minute. It would no longer hold a charge.

I have quite a lot of spare 12 volt 7Ah batteries that I can use as a replacement. I'm now using four of those batteries (older ones) in parallel.

Added wall charger as backup power (October 2020)

As we approached fall, the sun started to set earlier and earlier. The problem with my balcony is that I only have direct sunlight at 16:00 until sunset. My solar panel was therefore unable to keep the batteries charged.

I even added a smaller 60 watt solar panel I used for earlier tests in parallel to gain a few extra watts, but that didn't help much.

It is now at a point where I think it's reasonable to say that the project failed in my particular case. However, I do believe it would still be fine if I could capture the sun during the whole day (if my balcony wasn't in such a bad spot, the solar panel would be able to keep up).

As the batteries were draining I decided to implement a backup power solution, to protect the batteries. It's bad for lead acid batteries to be in a discharged state for a long time.

Therefore, I'm now using a battery charger that is connected to a relais that my software is controlling. If the voltage drops below 12.00 volt, it will start charging the batteries for 24 hours.

Upgraded Raspberry Pi 3b+ to a Raspberry Pi 4B (May 2022)

The idle power usage of the 4B is almost the same as the 3b+ model, although it requires some tweaking to reduce power usage.

The old 3b+ continuously complained about under voltage (detected), but the Pi4 seems to be less picky and it works fine with the XY-3606 power converter (12V to 5V).

the position of the panel is not optimal, so I will never get the panel's full potential. ↩
You don't have to buy the cable supplied by Victron, it's possible to create your own. The cable is not proprietary. ↩
It failed. Please read the update at the bottom of this article. ↩

Don't be afraid of RAID

2020-05-22T12:00:00+02:00

Introduction

I sense this sentiment on the internet that RAID is dangerous, that the likelihood of your RAID array failing during a rebuild is almost a certainty, because hard drives have become so large.

I think nothing is further from the truth and I would like to dispel this myth.

Especially for home users and small businesses, RAID arrays are still a reliable and efficient way of storing a lot of data in a single place.

Perception of RAID reliability

There are many horror stories to be found on the internet about people at home losing their RAID array. These stories may have contributed to a negative attitude towards RAID in general.

You may acuse me of victim blaming, but in many cases, I do wonder if those incidents were due to user error¹, due to bad luck or actual RAID causing problems. And there is a bias in reporting: you won't hear from the countless people who have no issues.

In any case, the damage is done, but I still think (software) RAID is perfectly fine.

The myth about the Unrecoverable Read Error (URE)

I think the trouble started with this terrible article on ZDNET from 2007.

In this article, it's argued that as drives become bigger, but not more reliable, you will see more unrecoverable read errors (UREs). More capacity means more sectors, so more risk of one of them going bad.

An URE is an incident where the hard drive can't read a sector⁵. For old people like me, that sounds like the definition of a 'bad sector'. The article argues that on average you would encounter an URE for every 12.5 TB of data read.

By the logic of the ZDNET acticle, just copying all data from a 14 TB drive would probably be impossible, because you would probably hit an URE / bad sector before you finish your copy.

This is a very big issue for RAID arrays. A RAID array rebuild consists of reading the contents of all remaining drives in their entirety². So you are guaranteed to hit an URE during a RAID rebuild.

The good news is that you don't have to worry about any of this. Because it is not true.

Hard drives are not that unreliable in practice. On the contrary. They are remarkably reliable, I would say. Just look at the Backblaze drive statistics⁶.

The prediction of the infamous ZDNET article has not come true. The URE specification for hard drive describes a worst-case scenario and seem to be more about marketing (a way to differentiate enterprise drives from consumer drives) than about reality.

If the ZDNET article were true, I, myself, should have encountered many UREs because of the many RAID array scrubs/patrol reads that have completed acros various RAID arrays.

RAID has never stopped working and is still going strong.

Card

Scrubbing protects against the impact of bad sectors

When a drive fails in a RAID array that can only tollerate one drive failure, it's very important that all remaining drives won't encounter any read errors. Because redundancy is lost, any read errors due to bad sectors could mean that the entire array is lost or at least some files are corrupted⁷.

Every RAID array supports 'scrubbing'. It's a process where every sector of the RAID array is read, which in effect causes all sectors of all hard drives to be read.

A scrub is a process to check for bad sectors in advance. If bad sectors are found on a hard drive, the drive can be replaced so it will not cause problems during a potential future rebuild. Replacing the drive itself will cause a rebuild, but assuming the scrub didn't find any other drives with bad sectors, that rebuild will be fine.

A RAID array that doesn't undergo a regular scrub is a disaster waiting to happen. Bad sectors may be building up on one of the other drivs and when a drive actually fails, the entire array may be lost because of the undetected bad sectors on (one of) the remaining drives.

If you want to store data in a reliable way on a RAID array, you need to assure the array is scrubbed periodically. And even if you don't use RAID, I would recommend running a long SMART test once a month against every hard drive you own.

By default, a Linux software RAID array is scrubbed once a week on Ubuntu. For details, look at the contents of /etc/cron.d/mdadm.

If you use ZFS on Linux, your array is automatically scrubbed on the second Sunday of every month if you run Ubuntu.

NAS vendors like Synology or QNAP have data scrubs enabled by default. Consider the manual of your particular NAS to adjust the frequency. I would recommend to scrub at least once a month and at night.

Why is RAID 5 considered harmful?

Frankly, I wonder that too.

I notice a lot of people on the internet claiming that you should never use RAID 5 but I disagree. It all depends on the circumstances. Finding a balance between cost and risk is important.

This page dating back to 2003 advocated not to use RAID 5 but that's focused on the enterprise environment and even there I see its uses.

For small RAID arrays with five or less drives I think RAID 5 is still a great fit. Especially if you run a small 4-bay NAS it would make total sense to use RAID 5. You get a nice balance between capacity and the cost of availability.

It's not really recommended to create larger RAID 5 arrays. Compared to a single drive, a RAID array with 8 drives is 8 times more likely to experience a drive failure. You multiply the risk of a single drive failing by eight. With larger arrays, double drive failure becomes a serious risk.

This is why it's really recommended to use RAID 6 for larger RAID arrays, because RAID 6 can tollerate two simultaneous drive failures. I've used RAID 6 in the past and I use RAIDZ2 (ZFS) as the basis for my current NAS.

I also run an 8-drive RAID 5 in one of my servers that hosts not so important data that I still want to keep around and would rather not lose, but not at every cost. It's all about a balance between risk and cost. Please also read the postscript of this post, you will like it.

It is true that during a rebuild, hard drives are strained more, but unless the RAID array is also in heavy use, the load on the drive isn't that big: the data is read sequentially, which is quite easy on the drives.

RAID rebuild performance is mostly determined by the size of the drives and not by the number of drives in the RAID array³.

Years ago I ran a 20-drive RAID 6 based on 1 TB drives and it did a rebuild in 5 hours. Recently I tested a rebuild of 8 drives in RAID 5 (using the same drives) and it also took almost 5 hours (4H45M).

The RAID write hole

The RAID 5/6 'write hole' is often mentioned as something you should be afraid about.

Parity-based RAID like RAID 5 and RAID 6 may be affected by an issue called the 'write hole'. To (over)simplify: if a computer would experience a sudden power failure, a write to the RAID array may be interrupted. This could cause a partial write to the RAID array, leaving it in an inconsistent state.

As a side note, I would always recommend protecting your NAS with a UPS (battery backup) so your server can shut down in a clean way, before power is lost as the battery runs out.

ZFS RAIDZ is not affected by the 'write hole' issue, because it writes data to a log first before writing it to the actual array⁴.

Linux MDADM software RAID also is protected against the 'write hole' phenomenon by using a bitmap (which is enabled by default⁴).

Hardware RAID is also protected against this by using a battery backup for the cache memory. The data in the cache memory is written to disk as soon as the computer is powered back on.

Setup alerting if you care about your data

I think that a lot of RAID horror stories are due to the fact that people may never notice any problems until it is too late because they never set up any kind of alerting (by email or other).

Ideally, you would also make sure your system monitors the SMART data of your hard drives and alert when critical numbers start to rise (Reallocated Sector count and Current Pending Sector count).

This is also a moment of personal reflection. Do you run a RAID array? Did you setup alerting? Or could your RAID array be failing this very moment and you wouldn't know?

Anyway: I think a lack of proper alerting is a nice way of getting into trouble with RAID, but that's not on RAID. Any storage solution that is not monitored is just a disaster waiting to happen.

Why people choose not to use RAID

If a RAID array fails, all data is lost. Some people are not comfortable with this risk. They would rather lose the contents of some drives, but not all of them.

Solutions like Unraid and SnapRAID use one or more dedicated hard drives to store redundant (parity) data. The other hard drives are formatted with your filesystem of choice and can be accessed as normal hard drives. Altough I have no experience with this product, StableBit DrivePool seems to work in a similar manner.

If you would have six hard drives, thus five data drives and one parity disk, the loss of two drives would result in data loss, as with RAID 5. However, the data on the remaining four drives would still be intact. The data loss is limited to just one drive worth of data.

The 'all-or-nothing' risk associated with regular software RAID is thus mitigated. I myself don't think those risks aren't that large, but Unraid and snapraid are popular product and I think they are reasonable alternatives.

Mergerfs could also be an interesting option, although it only supports mirroring.

Backups are still important

Storing your data on any kind of RAID array is never a substitute for a backup.

You should still copy your data to some other storage if you want to protect your data. You may chose to only make a backup of a subset of all of the data, but at least you take an informed risk.

Evaluation

I hope I have demonstrated why RAID is still a valid and reliable option for data storage.

Feel free to share your own views in the comments.

P.S.

I ran a scrub on my 8-disk RAID 5 array (based on 2 TB drives) as I was writing this article. My servers are only powered on when I need them and while powered off, it's easy for them to miss their periodic scrub window.

So as to practice what I preach I ran a scrub. Lo and behold, one of the drives was kicked out of my Linux software RAID array. Don't you love the irony?

sd 0:0:4:0: [sde] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:4:0: [sde] tag#29 Sense Key : Medium Error [current] 
sd 0:0:4:0: [sde] tag#29 Add. Sense: Unrecovered read error
sd 0:0:4:0: [sde] tag#29 CDB: Read(10) 28 00 9f 42 9e 30 00 04 00 00
print_req_error: critical medium error, dev sde, sector 2671943216

Followed by:

md/raid:md6: Disk failure on sde, disabling device.
md/raid:md6: Operation continuing on 7 devices.

The drive was clearly kicked out because the drive encountered bad sectors. A quick check of the SMART data revealed more than 300+ sectors were already remapped, but the data stored in them could not be recovered, causing read errors.

This drive is clearly done, although it was still operational.

After swapping this defective drive with a spare replacement, I started the rebuild proces, which took four hours and twenty minutes. My RAID 5 has rebuild and is now perfectly fine.

If an event like this doesn't drive the point home that scrubs are important, I don't know what will.

Sometimes I read what hardware people use for storage and I think about this quote by John Glenn: ‘I felt exactly how you would feel if you were getting ready to launch and knew you were sitting on top of 2 million parts — all built by the lowest bidder on a government contract.’ ↩
ZFS works differently, it only reads the sectors containing actual data. ↩
ZFS rebuilds or 'resilvers' become slower as you add more drives to a RAIDZ(2/3) VDEV, it seems. I'm not sure this is still the case with more recent ZFS versions. ↩
Both ZFS and MDADM will take a performance hit by using a log/bitmap. Both solutions support using an SSD to accelerate the log/bitmap to remove this performance hit. Most home users probably won't need this. ↩↩
The smallest unit of storage a drive can store, often 4K or 512 bytes for older, smaller drives. ↩
Those hard drive live in a datacenter with a conditioned environment, which you probably don't have at home. But as long as you keep the temperature of hard drive within limits, I don't think it matters that much. ↩
ZFS is both a RAID solution and a filesystem in one and can tell you exactly which file is affected. A nice feature. ↩

What home NAS builders should understand about silent data corruption

2020-04-23T12:00:00+02:00

Introduction

When it comes to dealing with storage in a DIY NAS context, two important topics come up:

Unrecoverable read errors (UREs) or what old people like me call 'bad sectors'
Silent data corruption (data corruption unnoticed by the storage layers)

I get a strong impression that people tend to confuse those concepts. However, they often come up when people evaluate their options when they want to buy or build their own do-it-yourself NAS.

In this article, I want to make a clear distinction between the two and assess their risk. This may help you evaluating these risks and make an informed decision.

Unrecoverable read errors (due to bad sectors)

When a hard drive hits a 'bad sector', it means that it can't read the contents of that particular sector anymore.

If the hard drive is unable to read that data even after multiple attempts, the operating system will return an Unrecoverable Read Error (URE).

This is an example (on Linux) of a drive experiencing read errors, as pulled from /var/log/syslog (culled a bit for readability):

sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current] 
sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error
sd 0:0:0:0: [sda] tag#19 CDB: Read(10) 28 00 02 1c 8c 00 00 00 98 00
blk_update_request: critical medium error, dev sda, sector 35425280 op 0x0:(READ)
sd 0:0:0:0: [sda] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda] tag#16 Sense Key : Medium Error [current] 
sd 0:0:0:0: [sda] tag#16 Add. Sense: Unrecovered read error
sd 0:0:0:0: [sda] tag#16 CDB: Read(10) 28 00 02 1c 8d 00 00 00 88 00
blk_update_request: critical medium error, dev sda, sector 35425536 op 0x0:(READ)

If a sector cannot be read, the data stored in that sector is lost. And in my experience, if you encounter a single bad sector, soon, there will be more. So if this happens, it's time to replace the hard drive.

We use RAID to protect against drive failure. RAID (no matter the implementation) also can deal with 'partial failure' such as a drive encountering bad sectors.

In a RAID array, a drive encountering unrecoverable read errors is just kicked out of the array, so it doesn't 'hang' or 'stall' the entire RAID array.

Please note that this behaviour does depend on the particular RAID solution of choice¹. The point is though that bad sectors or UREs are a common event and RAID solutions can deal with them properly.

The real problem with bad sectors (resulting in UREs) is that they can remain undiscovered until it is too late. So to uncover them in an early state, it's very important to run regular data scrubs. I've written an article specifically about this topic.

Silent data corruption

An unrecoverable read error means that we can't read (a portion of) a file. Although it is unfortunate - because we better have an intact backup of that file - we are also fortunate.

Why are we fortunate?

We are fortunate because the storage system - the hard drive and in turn the operating system - reported an error. We were able to diagnose the problem an take action.

But it is possible that bits and bytes get mangled without your hard drive, SATA controller or operating system noticing. Somewhere, somehow, a bit is read or transmitted as a 1 where it should have been a 0.

This is really bad, because this data corruption is undetected, it is 'silent', there is no notification.

Because imagine what happens: the corrupted file is happily backed up by your backup software, because it's unaware that anything is wrong. And by the time you discover the data corruption, the original pristine file is no longer part of the backup (rotated out). You are left with a lot of backups of a corrupted file. We encounter dataloss.

This is one of the scariest kinds of data loss. Because it's very difficult to detect. You'll have to constantly calculate the checksum of a file and verify it's still ok.

And that's - although rather simplified - exactly what ZFS does (amongst many other things). ZFS uses checksums at the block-level and thus assures with every read if the data contained in the block is still valid. ZFS is one of the few file systems that has this very powerfull feature (BTRFS is another example).

Regular RAID arrays (be it hardware-based or software-based) cannot detect silent data corruption (although it could be possible with RAID6). So it must be clear that ZFS is capable of protecting against a risk 'regular' RAID cannot cope with.

Is silent data corruption a significant threat for home DIY NAS builders?

Although silent data corruption is a very scary threat, from what I can tell there is no significant independant evidence that the risk of silent data corruption is so high that the average home DIY NAS builder should take this risk into account².

Maybe I'm wrong, but I think many people mistakenly confuse UREs or unrecoverable read errors (caused by bad sectors) with silent data corruption. And I think that's wrong, because there's nothing silent about an unrecoverable read error.

The truth is that hard drives are in fact very reliable when it comes to silent data corruption, because they make heavy use of error detection and correction algoritms. A significant portion of the raw capacity of a hard drive is sacrificed to store redundant information to aid in detecting and correcting data corruption. According to wikipedia, hard drives used Reed-Solomon error correction in the past and more modern drives use LDPC.

These error correction codes asure data integrity. Although 'soft' read errors may occur, there is enough additional redundant information stored on the hard drive to detect errors and even reconstruct the data (to some extend). Your hard drive handles this all by itself, it's part of normal operation.

So this is my point: it's important to understand that there is a lot of protection against silent data corruption in a hard drive. The risk of silent data corruption is therefore small³.

Sometimes the read data is so garbled that even the error correction codes cannot reconstruct the data as it was originally stored and that's what we then experience as an unrecoverable read error. But the disk notices! And it will report it!. This is not silent at all!

To really create silent data corruption, something very special need to happen. And to be very clear: such events do happen. But they are very rare.

Somehow, a bit must flip and this event is not detected by the error correction algorithm. Maybe the bit flipped in the hard drive cache memory when it was read from the drive. Maybe it flipped during transport over the SATA cable.

But it's fun to realise that the SATA protocol also has error detection embedded in the protocol for reliable data transmission. It's error detection and correction all the way down.

The risk that silent data corruption happens is thus very small, especially for home users.

Again, make no mistake: the risk is real and storage solutions for larger scale storage solutions (SANs / Storage arrays) with hundreds, thousands or tens of thousands of drives do really have to take into account the risk of silent data corruption. At scale, even very small risks become a certainty.

Enterprise storage solutions often employ their own proprietary solutions to protect against silent data corruption. Although it depend on the particular solution⁴, it's often part of the storage array. ZFS was revolutionary because they put the data integrity checking in the filesystem itself.

So if you think the risk of silent data corruption is still high enough that you should protect yourself against it, I would recommend to consider using ECC memory to protect against corrupted data in memory. To be frank: I consider non-ECC memory a more likely cause of silent data corruption than the storage subsystem, which already employs all these error detection and correction algoritms. Non-ECC memory is totally unprotected.

Anekdote: I myself run a 24-drive NAS based on ZFS and it has been rock-solid for 6 years straight.

From time to time, I do run disk 'scrubs', which can take quite some time. Although I have many terrabytes of data protected by ZFS, not a single instance of silent data corruption has been detected. And I have performed so many scrubs that I've read more than a petabyte worth of data.

Anekdote: Somebody made a mistake and used the wrong type of cable to connect the hard drives to the HBA controller card. This caused actual silent data corruption. Because that person was running ZFS, it was detected so ZFS saved his data. This an example where ZFS did protect a person against silent data corruption.

Evaluation

I hope that the difference between unrecoverable read errors and silent data corruption is clear and that we should not confuse the two. They have different risk profiles associated with them.

Furthermore, I have argued that silent data corruption is real and a serious issue at scale, and that it is that is dealt with accordingly.

However, I've also argued that unless you are a home user running a small datacenter inside your basement, the risk of silent data corruption is so small that it is reasonable to accept the risk as a DIY NAS builder and not seek specific protection against it.

The decision is up to you. If you want to go with ZFS and protect against silent data corruption, you should also be aware and accept the cost of ZFS. I myself have accepted that cost for my own NAS, but it's OK if you don't. If you care about silent data corruption so much, please also consider using ECC-memory.

But in my opinion, you are not taking an unreasonable risk if you chose to go with Unraid, Snapraid, Linux kernel RAID, Windows Storage Spaces or maybe other options in the same vein. I would say that this is reasonable and up to you.

Remember: the famous vendors of home user NAS boxes all seem to use regular Linux kernel RAID under the hood. And they seem to think that's fine.

In the end, what really matters is a solution that suits your needs and also fits your budget and level of expertise. Can you fix problems when something goes wrong?

I've noticed while testing with this particular drive that the drive was not kicked out of the array, and it just kept trying to read, grinding the Linux software RAID array to a halt. Removing the drive from the array fixed this. There is a 'failfast' option that only works with RAID1 or RAID10. ↩
I don't want to suggest in any way that it would be wrong to take silent data corruption into account, but just to say I think it's not mandatory to really fret over it. ↩
The most significant risk is that enterprise grade hard drives use on-board ECC cache memory, whereas consumer drives use non-ECC cache memory. So silently corrupted data in the cache memory of the drive could be a risk. ↩
Storage vendors often choose to reformat har drives with larger sector sizes⁵. Those larger sectors then also incorporate additional checksum data to better protect against data corruption or unrecoverable read errors. ↩
https://www.seagate.com/files/staticfiles/docs/pdf/whitepaper/safeguarding-data-from-corruption-technology-paper-tp621us.pdf ↩

Scrub your NAS hard drives regularly if you care about your data

2020-04-22T12:00:00+02:00

Introduction

Lots of people run a NAS at home. Maybe it's a COTS device from one of the well-known vendors¹, or it's a custom build solution (DIY²) based on hardware you bought and assembled yourself.

Buying or building a NAS is one thing, but operating it in a way that assures that you won't lose data is something else.

Obviously, the best way to protect against dataloss, is to make regular backups. So ideally, even if the NAS would go up in flames, you would still have your data.

Since backup storage costs money, people make tradeoffs. They may decide to take the risk and only backup a small portion of the really important data and take their chances with the rest.

Well that is their own right. But still, it would be nice if we would reduce the risk of dataloss to a minimum.

The risk: bad sectors

The problem is that hard drives may develop bad sectors over time. Bad sectors are tiny portions of the drive that have become unreadable³. How small a sector may be, if any data is stored in them, it is now lost and this could cause data corruption (one or more corrupt files).

This is the thing: those bad sectors may never be discovered until it is too late!

With todays 14+ TB hard drives, it's easy to store vast amounts of data. Most of that data is probably not frequently accessed, especially at home.

One or more of your hard drives may be developing bad sectors and you wouldn't even know it. How would you?

Your data might be at risk right at this moment while you are reading this article.

A well-known disaster scenario in which people tend to lose data is double hard drive failure where only one drive faillure can be tolerated (RAID 1 (mirror) or RAID 5, and in some scenario's RAID 10).

In this scenario, a hard drive in their RAID array has failed and a second drive (one of the remaining good drives) has developed bad sectors. That means effectively a second drive has failed although the drive may still seem operational. Due to the bad sectors, data required to rebuild the array is lost because there is no longer any redundancy⁴.

If you run a (variant of) RAID 5, you can only lose a single disk, so if a second disk fails, you lose all data⁵.

The mitigation: periodic scrubbing / checking of your disks

The only way to find out if a disk has developed bad sectors is to just read them all. Yes: all the sectors.

Checking your hard drives for bad sectors (or other issues) is called 'data scrubbing'. If you bought a NAS from QNAP, Synology or another vendor, there is a menu which allows you to control how often and when you want to perform a data scrub.

RAID solutions are perfectly capable of handling bad sectors. For a RAID array, it's just equivalent to a failed drive and an affected drive will be kicked out of the RAID array if bad sectors start causing read errors. The big issue we want to prevent is that multiple drives start to develop bad sectors at the same time, because that is the equivalent of multiple simultaneous drive failures, which many RAID arrays can't recover from.

For home users I would recommend checking all hard drives once a month. I would recommend configuring the data scrub to run at night (often the default) because a scrub may impact performance in a way that can be noticeable and even inconvenient.

Your vendor may have already configured a default schedule for data scrubs, so you may have been protected all along. If you take a look, at least you know.

People who have built a DIY NAS have to setup and configure periodic scrubs themselves or they won't happen at all. However, that's not entirely true: I've noticed that on Ubuntu, all Linux software RAID arrays (MDADM) are checked once a month at night. So if you use Linux software RAID you may already be scrubbing.

A drive that develops bad sectors should be replaced as soon as possible. It should no longer be trusted. The goal of scrubbing is to identify these drives as soon as possible. You don't want to get in a position that multiple drives have started developing bad sectors. You can only prevent that risk by scanning for bad sectors periodically and replacing bad drives.

You should not be afraid about having to spend a ton of money replacing drives all the time. Bad sectors are not that common. But they are a common enough that you should check for them. There is a reason why NAS vendors offer the option to run data scrubs and recommend them⁶.

You probably forgot to configure email alerting

If a disk in your NAS would fail, how would you know? If the scrub would discover bad sectors, would you ever notice⁷?

The answer may be: only when it's too late. Maybe a drive already failed and you haven't even noticed yet!

When you've finished reading this article, it may be the right moment to take some time to check the status of your NAS and configure email alerting (or any other alerting mechanism that works for you). Make your NAS sends out a test message just to confirm it actually works!

Closing words

So I would like to advice you to do two things:

Make sure your NAS runs a data scrub once a month
Make sure your NAS is able to email alerts about failed disks or scrubs.

These actions allow you to fix problems before they become catastrophic.

P.S. S.M.A.R.T. monitoring

Hard drives have a build-in monitoring system called S.M.A.R.T.

If you have a NAS from one of the NAS vendors, they will allert on SMART monitoring information that would indicate that a drive is failing. DIY builders may have to spend time setting up this kind of monitoring manually.

For more information about SMART I would recommend [this][this article] and this one.

this article and also this one

Linux users can take a look at the SMART status of their hard drives with this tool (which I made).

QNAP, Synology, Netgear, Buffalo, Thecus, Western Digital, and so on. ↩
FreeNAS, Unraid, Windows/Linux with Snapraid, OpenMediaVault, or a custom solution, and so on. ↩
Bad sectors cause 'unrecoverable read errors' or UREs. Bad sectors have nothing to do with 'silent data corruption'. There's nothing silent about unrecoverable read errors. Hard drives report read errors back to the operating system, they won't go unnoticed. ↩
A DIY NAS based on ZFS (FreeNAS is based on ZFS) may help mitigate the impact of such an event. ZFS can continue reading data from the remaining drives, even if bad sectors are encountered. Some files will be corrupted, but most of the data would still be readable. I think this capability is by itself not enough reason to pick a NAS based on ZFS because ZFS also has a cost involved that you need to accept too. For my large NAS I have chosen ZFS because I was prepared to 'pay the cost'. ↩
Some people may chose to go with RAID 6 which tolerates two simultaneous drive failures but they also tend to run larger arrays with more drives, which also increases the risk of drive failure or one of the drives developing bad sectors. ↩
Enterprise storage solutions (Even entry level storage arrays) often run patrol reads both on individual hard drives and also the RAID arrays on top of them. They are also enabled by default. ↩
At one time I worked for a small company that ran their own (single) email server. One of the system administrators discovered totally by accident that one of the two drives in a RAID 1 had failed. It turns out we were running on a single drive for months before we discovered it, because we forgot to setup email alerting. We didn't lose data, but we came close. ↩

Benchmarking storage with Fio and generating charts of the results

2020-04-21T12:00:00+02:00

Introduction

Fio is a widely-used tool for performing storage benchmarks. Fio offers a lot of options to create a storage benchmark that would best reflect your needs. Fio allows you to assess if your storage solution is up to its task and how much headroom it has.

Fio outputs .json and .log files that need further processing if you would like to make nice charts. Charts may help better communicate your test results to other people.

To make graphs of Fio benchmark data, I've created fio-plot. With fio-plot you can generate charts like:

It's very common that you want to run multiple benchmarks with different parameters to compare results. To generate the data of the charts, many benchmarks need to be run. This process needs to be automated.

Automating Fio benchmarks

I've chosen to build my own tool to automate Fio benchmarking. This tool is called bench_fio and is part of fio-plot. I'm aware that - as part of fio - a tool called genfio is provided, to generate fio job files with multiple benchmarks. It's up to you what you want to use. Bench-fio is tailored to output data in a way that aligns with fio-plot.

Bench-fio allows you to benchmark loads with different iodepths, simultaneous jobs, block sizes and other parameters. A benchmark run can consist of hundreds of tests and take many hours.

When you run bench_fio, you can expect output like this:

████████████████████████████████████████████████████
    +++ Fio Benchmark Script +++

Job template:                  fio-job-template.fio
I/O Engine:                    libaio
Number of benchmarks:          98
Estimated duration:            1:38:00
Devices to be tested:          /dev/md0
Test mode (read/write):        randrw
IOdepth to be tested:          1 2 4 8 16 32 64
NumJobs to be tested:          1 2 4 8 16 32 64
Blocksize(s) to be tested:     4k
Time per test (s):             60
Mixed workload (% Read):       75 90

████████████████████████████████████████████████████
4% |█                        | - [0:04:02, 1:35:00]-]

Bench-fio runs real-time and shows the expected remaining time. It also shows all relevant parameters that have been configured for this benchmark run. This makes it easier to spot any mis-configurations.

Notice that this benchmark consists of 98 individual tests: iodepth x NumJobs x Mixed Workload parameters (7 x 7 x 2). With a standard of 60 seconds per benchmark

This is an example of the command-line syntax: :::text ./bench_fio --target /dev/md0 -t device --mode randrw -o RAID_ARRAY --readmix 75 90

More examples can be found here.

My home network setup based on managed switches and VLANs

2020-04-10T12:00:00+02:00

My home networking setup

I live in a two story apartment, with on the top floor my utilities closet and my living room. The bottom floor contains a bedroom with all my servers and networking gear.

So this is my setup (click for a bigger version):

I like to run my own router but my utilities closed is not the safest place in terms of security and climate.

By default, most people who run their own home router will use a box with two network interfaces, one connected to the (cable) modem and the other one connected to the home network.

I could have done the same thing, by running a cable from the modem to my router, and a second cable back up towards my closet (and livingroom).

However, I didn't want to run multiple cables from my utilities closed to my bedroom downstairs, I saw no need for that: because I can use VLANs.

The small 8-port switch in the closed is connected with a single (long) cable to the 24-port switch I have in my bedroom downstairs. This switch connects to my router and multiple servers.

I've setup a trunk between these two switches where my internet traffic flows over 'VLAN 100' and my home network uses 'VLAN 200'.

The router, an HP N40L, has only a single network interface. I just expose the two VLANS as 'tagged' and let the router route traffic between the two VLANS. No need for a second interface (as many home setups do).

So in my setup there are two trunks, one between the two switches and the other one between the bedroom switch and my router. All other devices are connected to untagged network ports, in their appropriate VLAN.

The small switch in the closet is responsible for carrying my home network to the switch in my living room.

The raspberry pi connects to my smart meter to collect information about my power and gas usage.

The impact of the MDADM bitmap on RAID performance

2020-04-06T12:00:00+02:00

Introduction

I'm aware that most people with intensive storage workloads won't run those workloads on hard drives anymore, that ship has sailed a long time ago. SSDs have taken their place (or 'the cloud').

For those few left who do use hard drives in Linux software RAID setups and run workloads that generate a lot of random IOPS, this may still be relevant.

I'm not sure how much a bitmap affects MDADM software RAID arrays based on solid state drives as I have not tested them.

The purpose of the bitmap

By default, when you create a new software RAID array with MDADM, a bitmap is also configured. The purpose of the bitmap is to speed up recovery of your RAID array in case the array gets out of sync.

A bitmap won't help speed up the recovery from drive failure, but the RAID array can get out of sync due to a hard reset or power failure during write operations.

The performance impact

During some benchmarking of various RAID arrays, I noticed very bad random write IOPS performance. No matter what the test conditions were, I got the random write performance of a single drive, although the RAID array should perform better.

Then I noticed that the array was configured with a bitmap. Just for testing purposes, I removed the bitmap all together with:

mdadm --grow --bitmap=none /dev/md0

Random write IOPs figures improved immediately. This resource explains why:

If  the  word internal is given, then the bitmap is stored with the metadata
on the array, and so is replicated on all devices.

So when you write data to our RAID array, the bitmap is also constantly updated. Since that bitmap lives on each drive in the array, it's probably obvious that this really deteriorates random write IOPS.

Some examples of the performance impact

Bitmap disabled

An example of a RAID 5 array with 8 x 7200 RPM drives.

Another example with 10.000 RPM drives:

Bitmap enabled (internal)

We observe significant lower random write IOPs performance overall:

Which is also true for 10.000 RPM drives.

External bitmap

You could keep the bitmap and still get great random write IOPS by putting the bitmap on a separate SSD. Since my boot device is an SSD, I tested this option like this:

mdadm --grow --bitmap=/raidbitmap /dev/md0

I noticed excellent random write IOPS with this external bitmap, similar to running without a bitmap at all. An external bitmap has it's own risks and caveats, so make sure it really fits your needs.

Note: external bitmaps are only known to work on ext2  and  ext3. 
Storing bitmap files on other filesystems may result in serious problems.

Conclusion

For home users who build DIY NAS servers and who do run MDADM RAID arrays, I would recommend leaving the bitmap enabled. The impact on sequential file transfers is negligible and the benefit of a quick RAID resync is very obvious.

Only if you have a workload that would cause a ton of random writes on your storage server would I consider disabling the bitmap. An example of such a use case would be running virtual machines with a heavy write workload.

Update on bitmap-chunks

Based on feedback in the comments, I've performed a benchmark on a new RAID 5 array setting the --bitmap-chunk option to 128M (Default is 64M).

The results seem to be significantly worse than the default for random write IOPS performance.

Cryptocurrencies are detrimental to society

2020-03-27T12:00:00+01:00

Want to listen to this article?

Your browser does not support the audio element.

Introduction

How would you explain the inner workings of bitcoin to a person in simple, understandable terms?

source

This explanation seems perfect to me because it illustrates some seriously problematic aspects of cryptocurrencies in one simple sentence.

It captures the unimaginable energy waste of mining cryptocurrencies. And it also captures the dark side of cryptocurrencies: facilitating crime.

At this point cryptocurrency enthusiasts are rolling their eyes and sigh. This point has been made many times over. I know, I'm definitely not the first to criticise cryptocurrencies¹ this way.

I do have a simple challenge though:

Can you please show me the benefit to society of cryptocurrencies?

Please, don't come up with theoretical or future possibilities. Cryptocurrencies have existed for eleven years, they should have something to show for right now. After many hours of reading up on the topic, I have not been able to find any tangible benefits that would justify the effort and resources spend on them.

The downsides of cryptocurrencies are an entirely different matter. They are very, very clear to me. But let's not go there directly. What fun would that be?

The solution in search of a problem

As I see it, cryptocurrencies are entangled in a desparate search for a problem to solve. They are the answer to a question nobody asked².

Many cryptocurrency advocates see the decentralised (distributed) nature of cryptocurrencies as a tangible benefit. Cryptocurrencies are most often not controlled by any government or single entity.

Aside from whether this is true in practice³, they seem to imply that it's a bad thing that governments control their own currencies. Well, last time I checked governments control their currencies to keep them as stable as possible. Frankly, that actually sounds like exactly what I would want from a currency.

Stability.

You can call cryptocurrencies many things but 'stable' is definitely not one of them. Cryptocurrencies are highly volatile. In cryptocoin, a loaf of bread could suddenly cost twice as much as the day before.

Although cryptocurrencies haven't seen any significant adoption as a payment method - due to their volatility - it has seen adoption in 'less stable countries' where it is basically the 'lesser of two evils'. I mean: you know things are bad if a volatile cryptocurrency is a safer option than the native currency of your country.

The argument in the end boils down to: if your society is already in serious trouble, maybe cryptocurrencies could provide 'some benefit'. If the people involved have reliable access to internet. And internet access is most often controlled by the government.

To my knowledge, cryptocurrency as a payment method has actually only seen true adoption within the world of dark markets such as Silk Road in wich bitcoin rose to prominence. Nonetheless, after eleven years, cryptocurrencies have no traction in the regular 'legitimate' markets as a real payment method.

The reason why is obvious: existing payment methods are much easier to use and feel much safer. And the volatility of cryptocurrencies only compounds to the support of these conventional methods.

If cryptocurrencies are a solution to anything at all they seem to be 'bad' solutions at best.

Are cryptocurrencies in fact a Ponzi or Piramid scheme?

It depends on the particular currency, but I think the case can be made for sure. I mean: why not both?

Cryptocurrencies have no intrinsic value. They are only worth what people are willing to pay for them. So the cryptocurrency advocates needed to drum up demand, to create a market where previously none existed. This resulted in wild visions of the future, elaborate jargon-filled smokescreens that argue cryptocurrencies would take over the world. Get in quickly or you miss out!

And so many people were afraid to miss out, creating an enormous cryptocurrency hype, starting in November of 2017, spilling into 2018 when the bubble burst. The end result? A few people got very rich and the vast majority lost money.

It seems to me this all is a combination of a ponzi scheme with the component of active recruitment found in piramid schemes. The value of the currencies must come from somewhere, right?

So please tell me how all of this benefits our society? Creating a handful of rich people at the expense of a lot of other people? Is that it?

The graveyard of dead cryptocurrencies only shows how many people or startups try to get a piece of the action. And so many of them are outright scams. Are in fact all cryptocurrencies scams at their core?

The downsides of cryptocurrencies

I hope I have established that cryptocurrencies provide no tangible benefits to society. But the do have a lot of downsides. I observe the following:

Topic	Remark
Trafficing of illegal goods	Drugs, weapons, childpornography, and so on.
Trafficing of illegal services	Murder for hire
Tax evasion	-
Money laundering	-
Ransomware	Hold data hostage in encrypted form
Polution / energy waste	crypto miners use a lot of electricity
Lack of Security	several cryptocurrency exchanges have been hacked
Crypto scams	New cryptocoins are created just to scam people

The easy retort to this table is: "'normal' currencies like the dollar or euro also facilitate almost all of those illegal things", which is true but it misses the larger point.

Those regular currencies provide tremendous value to our societies. Our societies are build upon them and they facilitate almost everything we do. The topics listed in the table are just a possible negative side-effect for regular currencies. Their clear benefits outweighs such downsides.

Cryptocurrencies don't seem to have any such upsides. They seem to be made to exclusively facilitate cryptocurrency speculation and crime.

Cryptocurrencies facilitate crime

I won't discuss all the topics in the previous table but I do want to highlight a few.

Dark markets

I think we all remember Silk Road, a now defunct darknet marketplace that allowed people to anonymously buy - amongst other things - drugs and guns⁴. Silk Road was the first large-scale application of bitcoin as a means of payment.

Silk Road started out with just drugs, but guns soon followed. This deeply depraved world of dark markets are very much enabled by cryptocurrencies because the parties involved in a transaction are so hard to identify.

Ransomware

Ransomware seems to be almost exclusively enabled by digital currencies. Not explicitly just cryptocurrencies, but they do enable this type of crime because tracing the payments back to the criminal is so difficult.

The damage caused by ransomware is so obviously devastating. You can have many opinions on the fact that many critical organisations such as hospitals or universities don't have their computer security under control.

The real problem is that cryptocurrencies make these kinds of attacks on businesses and institutions very low-risk and highly profitable. In my own country a university was targeted by such an attack and allegedly they paid the ransom. The disruption to its services was substantial.

Energy waste

As cryptocurrencies rose in value, it started to become profitable to 'mine' them. It started out with regular computers, but soon, we could use videocards to accelerate cryptocurrency mining, which are very power hungry.

Later on, FPGAs and ASICS were build to further accelerate mining performance. Entire companies spun up to build those miners and host them in large datacenters with cheap electricity. The scale of the operation is rather enormous.

According to an article dating to July 2019, just bitcoin mining consumes more electricity than Switzerland. The article links to an online tool that tracks this power usage in real-time based on some estimates.

Image source

It's just mind boggling to me that so much energy is wasted, so much pressure is put on the environment, for absolutely no clear benefit at all.

Closing words

So in short, I think that cryptocurrencies provide nothing of value to society. They do however facilitate crime and contribute to climate change.

Therefore, I would propose to shut them all down.

The complex technology behind the cryptocurrencies attracted a lot of otherwise smart people and I think it's a sad thing to see their efforts going to waste or have a negative impact.

People are not obliged to work on something valuable, but at least may I ask that they choose to work on something that won't harm our society?

Link to hackernews, where this post was quickly flagged down. The few comments that exist don't seem to really provide any answers to the question I pose.

I would definitely recommend reading this long-form-article by The New York Times. ↩
Unless you want to embark on a path of criminal activity. ↩
As mining is no longer profitable for the larger community, the miners become a small concentrated group of entities controlling the currency, making the currencies more centralised. Furthermore, the cryptocurrency exchanges where you can convert the cryptocurrency into regular money, are centralised institutions backed by for-profit companies. And those companies have to abide by the law. They are under the influence of the government. ↩
If you want to know more about what happend to Silk Road and it's founder - 'the Dread Pirate Roberts', I would recommend the book 'American Kingpin' by Nick Bilton. (no affiliate links) ↩

Understanding Storage Performance - IOPS and Latency

2020-03-21T12:00:00+01:00

Introduction

The goal of this blogpost is to help you better understand storage performance. I want to discuss some fundamentals that are true regardless of your particular needs.

This will help you better reason about storage and may provide a scaffolding for further learning.

If you run your applications / workloads entirely in the cloud, this information may feel antiquated or irrelevant.

However, since the cloud is just somebody else's compute and storage, knowledge about storage may still be relevant. Cloud providers expose storage performance metrics for you to monitor and this may help to make sense of them.

Concepts

I/O

An I/O is a single read/write request. That I/O is issued to a storage medium (like a hard drive or solid state drive).

It can be a request to read a particular file from disk. Or it can be a request to write some data to an existing file. Reading or writing a file can result in multiple I/O requests.

I/O Request Size

The I/O request has a size. The request can be small (like 1 Kilobyte) or large (several megabytes). Different application workloads will issue I/O operations with different request sizes. The I/O request size can impact latency and IOPS figures (two metrics we will discuss shortly).

IOPS

IOPS stands for I/O Operations Per Second. It is a performance metric that is used (and abused) a lot in the world of storage. It tells us how many I/O requests per second can be handled by the storage (for a particular workload).

Warning: this metric is meaningless without a latency figure. We will discuss latency shortly.

Bandwidth or throughput

If you multiply the IOPS figure with the (average) I/O request size, you get the bandwidth or throughput. We state storage bandwidth mostly in Megabytes and Gigabytes per second.

To give you an example: if we issue a workload of 1000 IOPS with a request size of 4 Kilobytes, we will get a throughput of 1000 x 4 KB = 4000 KB. This is about ~4 Megabytes per second.

Latency

Latency is the time it takes for the I/O request to be completed. We start our measurement from the moment the request is issued to the storage layer and stop measuring when either we get the requested data, or get confirmation that the data is stored on disk.

Latency is the single most important metric to focus on when it comes to storage performance, under most circumstances.

For hard drives, an average latency somewhere between 10 to 20 ms is considered acceptable (20 ms is the upper limit).

For solid state drives, depending on the workload it should never reach higher than 1-3 ms. In most cases, workloads will experience less than 1ms latency numbers.

IOPS and Latency

This is a very important concept to understand. The IOPS metric is meaningless without a statement about latency. You must understand how long each I/O operation will take because latency dictates the responsiveness of individual I/O operations.

If a storage solution can reach 10,000 IOPS but only at an average latency of 50 ms that could result in very bad application performance. If we want to hit an upper latency target of 10 ms the storage solution may only be capable of 2,000 IOPS.

For more details on this topic I would recommend this blog and this blog.

Access Patterns

Sequential access

An example of a sequential data transfer is copying a large file from one hard drive to another. A large number of sequential (often adjacent) datablocks is read from the source drive and written to another drive. Backup jobs also cause sequential access patterns.

In practice this access pattern shows the highest possible throughput.

Hard drives have it easy as they don't have to spend much time moving their read/write heads and can spend most time reading / writing the actual data.

Random access

I/O requests are issued in a seemingly random pattern to the storage media. The data could be stored all over various regions on the storage media. An example of such an access pattern is a heavy utilised database server or a virtualisation host running a lot of virtual machines (all operating simultaneously).

Hard drives will have to spend a lot of time moving their read/write heads and can only spend little time transferring data. Both throughput and IOPS will plummet (as compared to a sequential access pattern).

In practice, most common workloads, such as running databases or virtual machines, cause random access patterns on the storage system.

Queue depth

The queue depth is a number between 1 and ~128 that shows how many I/O requests are queued (in-flight) on average. Having a queue is beneficial as the requests in the queue can be submitted to the storage subsystem in an optimised manner and often in parallel. A queue improves performance at the cost of latency.

If you have some kind of storage performance monitoring solution in place, a high queue depth could be an indication that the storage subsystem cannot handle the workload. You may also observe higher than normal latency figures. As long as latency figures are still within tolerable limits, there may be no problem.

Storage Media Performance characteristics

Hard drives

Hard drives (HDDs) are mechanical devices that resemble a record player.

They have an arm with a read/write head and the data is stored on (multiple) platters.

Hard drives have to physically move read/write heads to fulfil read/write requests. This mechanical nature makes them relatively slow as compared to solid state drives (which we will cover shortly).

Especially random access workloads cause hard drives to spend a lot of time on moving the read/write head to the right position at the right time, so less time is available for actual data transfers.

The most important thing to know about hard drives is that from a performance perspective (focussing on latency) higher spindle speeds reduce the average latency.

Rotational Speed (RPM)	Access Latency(ms)	IOPS
5400	17-18	50-60
7200	12-13	75-85
10,000	7-8	120-130
15,000	5-6	150-180

Because the latency of individual I/O requests is lower the drives with a higher RPM, you can issue more of such requests in the same amount of time. That's why the IOPS figure also increases.

Latency and IOPS of an older Western Digital Velociraptor 10,000 RPM drive:

Notice the latency and IOPS in the Queue Depth = 1 column.

Source used to validate my own research.

Regarding sequential throughput we can state that fairly old hard drives can sustain throughputs of 100-150 megabytes per second. More modern hard drives with higher capacities can often sustain between 200 - 270 megabytes per second.

An important note: sequential transfer speeds are not constant and depend on the physical location of the data on the hard drive platters. As a drive fills up, throughput diminishes. Throughput can drop more than fifty percent! ¹.

So if you want to calculate how long it will take to transfer a particular (large) dataset, you need to take this into account.

Solid State Drives

Solid state drives (SSDs) have no moving parts, they are based on flash memory (chips). SSDs can handle I/O much faster and thus show significantly lower latency.

Whereas we measure the average I/O latency of HDDs in milliseconds (a thousand of a second) we measure the latency of SSD I/O operations in microseconds (a millionth of a second).

Because of this reduced latency per I/O request, SSDs outperform HDDs in every conceivable way. Even a cheap consumer SSD can at least sustain about 5000+ IOPS with only a 0.15 millisecond (150 microseconds) latency. That latency is about 40x better than the best latency of an enterprise 15K RPM hard drive.

Solid state drives can often handle I/O requests in parallel. This means that larger queue depths with more I/O requests in flight can show significantly higher IOPS with a limited (but not insignificant) increase in latency.

The random I/O performance of an older SATA consumer SSD

More modern enterprise SSDs show better latency and IOPS. The SATA interface seems the main bottleneck.

The random I/O performance of an enterprise SATA SSD

SSDs perform better than HDDs across all relevant metrics except price in relation to capacity.

Important note: SSDs are not well-suited for archival storage of data. Data is stored as charges in the chips and those charges can diminish over time. It's expected that even hard drives are better suited for offline archival purposes although the most suitable storage method would probably be tape.

SSD actual performance vs advertised performance

Many SSDs are advertised with performance figures of 80,000 - 100,000 IOPS at some decent latency. Depending on the workload, you may only observe a fraction of that performance.

Most of those high 80K-100K IOPS figures are obtained by benchmarking with very high queue depths (16-32). The SSD benefits from such queue depths because it can handle a lot of those I/O requests in parallel.

Please beware: if your workload doesn't fit in that pattern, you may see lower performance numbers.

If we take a look at the chart above of the Intel SSD we may notice how the IOPS figures only start to come close to the advertised 80K+ IOPS as the queue depth increases. It's therefore important to understand the characteristics of your own workload.

RAID

If we group several hard drives together we can create a RAID array. A RAID array is a virtual storage device that exceeds the capacity and performance of a single hard drive. This allows storage to scale within the limits of a single computer.

RAID is also used (or some say primarily used) to assure availability by assuring redundancy (drive failure won't cause data loss). But for this article we focus it's performance characteristics.

SSDs can achieve impressive sequential throughput speeds, of multiple gigabytes per second. Individual hard drives can never come close to those speeds, but if you put a lot of them together in a RAID array, you can come very close. For instance, my own NAS an achieve such speeds using 24 drives.

RAID also improves the performance of random access patterns. The hard drives in a RAID array work in tandem to service those I/O requests so a RAID array shows significantly higher IOPS than a single drive. More drives means more IOPS.

RAID 5 with 8 x 7200 RPM drives

The picture below shows the read IOPS performance of an 8-drive RAID 5 array of 1 TB, 7200 RPM drives. We run a benchmark of random 4K read requests.

Notice how the IOPS increases as the queue depth increases.

However, nothing is free in this world. A higher queue depth - which acts as a buffer - does increase latency.

Notice how quickly the latency exceeds 20ms and quickly becomes almost unusable.

RAID 5 with 8 x 10,000 RPM drives

Below is the result of a similar test with 10,000 RPM hard drives. Notice how much better the IOPS and latency figures are.

The latency looks much better:

It makes sense to put SSDs in RAID. Although they are more reliable than hard drives, they can fail. If you care about availability, RAID is inevitable. Furthermore, you can observe the same benefits as with hard drives: you pool resources together, achieving higher IOPS figures and more capacity than possible with a single SSD.

Capacity vs. Performance

The following is mostly focussed on hard drives although it could be true for solid state drives as well.

We put hard drives in RAID arrays to get more IOPS than a single drive can provide. At some point - as the workload increases - we may hit the maximum number of IOPS the RAID array can sustain with an acceptable latency.

This IOPS/Latency threshold could be reached even if we have only 50% of the storage capacity of our RAID array in use. If we use the RAID array to host virtual machines for instance, we cannot add more virtual machines because this would cause the latency to rise to unacceptable levels.

It may feel like a lot of good storage space is going to waste, and in some sense this may be true. For this reason, it could be a wise strategy to buy smaller 10,000 RPM or 15,000 RPM drives purely for the IOPS they can provide and forgo on capacity.

So it might be the case that you may have to order and add let's say 10 more hard drives to meet the IOPS/Latency demands while there's still plenty of space left.

This kind of situation is less likely as SSDs have taken over the role of the performance storage layer and (larger capacity) hard drives are pushed in the role of 'online' archival storage.

Closing words

I hope this article has given you a better understanding of storage performance. Although it is just an introduction, it may help you to better understand the challenges of storage performance.

https://en.wikipedia.org/wiki/Hard_disk_drive_performance_characteristics#Data_transfer_rate ↩

Difference of behavior in SATA Solid State Drives

2020-01-29T00:00:00+01:00

Introduction

Update: I've noticed some strange behavior of SSDs when benchmarking them with FIO. After further investigation and additional testing, I've found the reason for the strange patterns in the graphs.

The 'strange' test results are due to the fact that they were obtained by connecting the SSDs to a P420I controller. As the HBA mode of this controller performs worse than the RAID mode, I used the RAID mode of this controller. Indvidual drives were put in a RAID0 volume. But it turns out that this creates a strange interaction between RAID controller and SSD.

Additional testing with an SATA 300 AHCI controller shows 'normal' patterns that look similar to the results of the INTEL SSD as compared to the other ones (Samsung and Kingston).

It seems I've made a mistake by using the P420i controller for testing. I have includes both 'bad' and 'good' results.

Regular SATA solid state drives may seem interchangeable at this point. They all show amazing IOPS and latency performance.

I have performed benchmarks on different SSDs from different vendors and it seems that they actually show very different behaviour. This behavior has come to light because I benchmarked the entire device capacity.

The benchmark - performed with FIO - puts a fifty percent read/write random 4K workload on the device. The benchmark stops when all sectors of the device have been read or written to. Furthermore, all tests are performed with a queue depth of 1.

I've made this post because I found the results interesting. At least the images show a very peculiar pattern for some SSDs. I can't explain them really, maybe you can.

This is the test I ran against the SSDs.

fio --filename=/dev/sdX --direct=1 --rw=randrw --refill_buffers
--norandommap --ioengine=libaio --bs=4k --rwmixread=50 --iodepth=1

Disclaimer

I've performed these benchmark to the best of my knowledge. The raw benchmark data is available here

It's always possible that I made a mistake, so it may be wise to run your own tests to see if you can replicate these results.

Caveat: I really don't know if these benchmark results impact real-life performance. Maybe these benchmark results show a kind of behaviour of SSDs that doesn't really matter in the end.

Benchmark Results

Intel D3-S4610

This SSD is meant for for datacenter usage. This is the test result on the P420i controller.

IOPS and Latency is consistent during the whole benchmark. It's behaviour seems predictable.

This is the test result on the AHCI controller:

Samsung 860 Pro

This SSD is meant for desktop usage. Its behavior seems quite different from the Intel SSD. I have separated the IOPS data from the Latency data to make the graphs more eligible.

This is the test result on the P420i controller.

IOPS

Latency

The best-case latency is almost four times better than the worst-case latency. Latency is thus less predictable. This impact also seems to be reflected in the IOPs numbers.

This is the test result on the AHCI controller:

Samsung PM883

This SSD is meant for datacenter usage. This is the test result on the P420i controller.

IOPS

Latency

This SSD seems to behave in a similar way as the 860 PRO.

This is the test result on the AHCI controller:

Kingston DC500M

This SSD is meant for datacenter usage. This is the test result on the P420i controller.

IOPS

latency

The behavior of this SSD seems similar to the behaviour of the Samsung SSDs but the pattern is distinct: it seems shifted as compared to the Samsung SSDs.

This is the test result on the AHCI controller:

Evaluation

** Updated evaluation**

We can conclude that the P420i RAID controller causes strange behavior not observed when we test the SSDs on a regular AHCI controller. Although this was an older SATA 300 controller, I'm making the assumption that this controller still has enough bandwidth to support a random 4K test as most tests never went beyond 50+ MB/s of throughput.

At this point, I can only say that I observe quite different behavior between the Intel SSD and the other SSDs from Samsung and Kingston. The problem is that I can't tell if this affects real-life day-to-day application performance.

It seems that although results for the Samsung and Kingston SSDs fluctuate quite a bit, it's quite possible that the fluctuations occur during a very short timespan and effectively cancel each other out.

If you have comments, ideas or suggestions, leave a comment below.

How are these images generated?

All images have been generated with fio-plot.

The github repository also contains a folder with a lot of example images.

Benchmarking a 1.44 MB floppy drive

2019-12-17T00:00:00+01:00

I've used fio - a storage benchmarking tool - to benchmark a 1.44 MB floppy drive to gauge its random I/O performance. I have no real reason for doing this. I just thought it was funny.

I've run a benchmark against the floppy drives for different queue depths. The results are shared below.

Random 4K Read performance

Random 4K Write performance

Conclusion

These images show that 1.44 MB a floppy drive have atrocious random I/O performance as compared to hard drives or solid state drives. Who knew!

Next: benchmarking random I/O performance of a tape drive. If somebody will lend me one.

Tools used

I used fio to benchmark the floppy drive. I've used fio-plot to generate the images. The benchmarks were run using an altered version of the benchmark script included with fio-plot.

Fio-plot: creating nice charts from FIO storage benchmark data

2019-11-28T00:00:00+01:00

New release of fio-plot

I've rewritten fio-plot, a tool that turns FIO benchmark data into nice charts. It allows you to create four types of graphs which will be discussed below.

The github project page explains how to run this tool.

Fio-plot also includes a benchmark script that automates testing with Fio.

The git repository also contains benchmark data, which can be used to test fio-plot.

2D bar chart

This chart plots IOPs and latency for various queue depths.

3D bar chart

2D line chart for log data

2D bar chart latency histogram

Additional images

The github repository also contains a folder with a lot of example images.

My home lab server with 20 cores / 40 threads and 128 GB memory

2019-08-13T00:00:00+02:00

I've bought a secondhand HP DL380p Gen8 server and I think it's awesome.

This machine has dual-processors, with each CPU having 10 physical cores and two threads per core, for a grand total of 20 cores / 40 threads for the entire machine. It's also equiped with 128 GB or memory.

Laughs in htop

I bought this machine because I wanted to have a dedicated server on which I run my home lab environment. This is the box on which I will try out new software, run experiments, test ideas and so on.

In this article I'd like to share some information about this machine and my experiences with it. Let's start with an overview of the specifications.

Update: this article featured on Hacker News, click here for the discussion thread.

HP DL380p Gen8 Specifications

Part	Description
Form factor	19" 2U rackmount
Processor	2 x Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
RAM	128 GB DDR3 (16 x 8GB)
Onboard LAN	4 x Broadcom 1 Gigabit (Copper)
RAID Controller	HP P420i
Storage	8 x 2.5" SAS/SATA slots (no media)
PSU	1 x 750 Watt
Power usage	about 110 Watt idle
KVM	HP iLO 4 with HTML5 console

Cost

I've bought this server for around 1500 Euro ($1700) including taxes (21%) and a one year warranty. I've bought it from the company https://creoserver.com (The Netherlands).

Update 2021: This kind / generation of hardware seems to be dumped on the market for less than half of what I paid for it at the time.

I believe that the price/performance balance for this server is quite good, but I let you be the judge of that.

The storage is based on some SSDs I already owned and some new SSDs I've bought separately. So the price is based on the chassis + storage caddies but without actual storage media.

Age

The HP DL380p Gen8 product generation was introduced in 2012. The configuration I have chosen is based on a CPU that was introduced in 2013. Based on the serial number I guess the chassis+motherboard is from 2014 (about 5 years old¹ at the time I bought it).

CPU - The Intel Xeon E5-2680 v2

The Intel Xeon E5-2680 v2 CPU originated from late 2013 and is based on the ivy bridge architecture. The base clock is 2.8 Ghz but it can turbo boost up to 3.6 Ghz. With 10C/20T per CPU and the dual-processor configuration it's needless to say that you can run quite a few virtual machines and/or containers in parallel on this machine, without even oversubscribing CPU cores.

You may ask what kind of performance you may expect from this box and how 2013 holds up in 2019. A high number of cores is nice, but the cores themselves fast enough?

The 2680v2 is a processor that sold for about $1700 (Intel Ark) when it came out in 2013. If you look at the list of Xeon processors from that timeframe, it was the top of the line Intel dual-processor compatible CPU with only the E5-2690 v2 processor above it, which ran only 200 Mhz faster (base clock).

There was almost nothing better at that time and people ran their companies on those machines so I think performance is more than enough. Especially for a lab environment (overkill is probably an understatement).

I've included some benchmark results below just for reference. You can use this information to make some comparisons to other CPUs that you are familiar with to get some sense of the performance.

I've also compared scores a little bit and I've noticed that people in the past were perfectly happy gaming on desktop systems with CPUs that have equal or less single core performance.

Geekbench 4 score

I've run Geekbench on the DL380p Gen8:

Single Core:  3525
Multi Core : 48764

Passmark

(source)

Single Core:  1788
Multi Core : 12612

Sysbench

root@dora:/home/ansible# sysbench --test=cpu --cpu-max-prime=20000 run
[.....]
General statistics:
    total time:                          10.0002s
    total number of events:              4358
[.....]

Some perspective: My intel Mac mini 2018 and my game pc from 2013 also hit 10 seconds runtime. For whoever thinks they should work with 'cheap' Raspberry Pi's, this is the sysbench score of a Raspberry Pi 3B+.

Test execution summary:
    total time:                          328.3016s

It's 32x slower. I really recommend to not use Pi's except for electronic projects. The performance is so bad, it's not worth it. Even a small VM on your desktop/laptop is already way faster.

Other benchmarks

I won't promise anything, but if it's not too much work I can run a benchmark if you have any suggestions. Feel free to comment down below or email me.

Memory

I've ordered this machine with 128GB of DDR3 memory. The memory consists of 16 x 8GB 1333Mhz memory modules. The machine has 12 memory slots per CPU so there are still 8 slots left to be filled if I ever feel a need to expand memory.

Storage

Drive slots

I've chosen a chassis with room for 8 x 2.5" drives. Depending on the chassis model, you can go up to 25 x 2.5" (Small Form Factor) or 12 x 3.5" (Large Form Factor).

Both SAS and SATA HDDs/SSDs are supported. You can't just stick HDDs or SSDs in the drive slots, you need to put your HDD/SSD first in an empty caddy, I bought those with the server.

The caddy has the LED lights build-in to signal the drive status. It has some electrical contacts at the back for those lights, it's not just a plastic 'sabot'.

My caddies tended to be a bit fragile, they came apart when trying to pull a drive out of the cage, but nothing broke and it all works fine.

Drive compatibility

HP servers can be picky about which HDD disk drives work or won't work. They all technically work, but sometimes the disk temperature cannot be read by the controller. In that case, the system fans will slowly start to spin faster and faster until almost maximum speed. This is obviously not workable.

Smaller HDD drives did work fine and the server stayed quiet. Only some 500GB HDD drives caused 'thermal runaway'.

I've found this table of supported and unsupported drives for HP controllers.

Overview of SSDs that are compatible

I've tested these SSDs:

- Samsung 860 Pro (consumer)
- Samsung PM833   (enterprise)
- Intel D3-S4610  (enterprise)
- Crucial MX200   (consumer)

P420i RAID mode vs. HBA mode

Important: you can't boot from the storage controller in HBA mode. You are left with booting from internal/external USB (2.0) or internal SD card.

I'm not sure if there is any significant performance difference between the two modes for SSDs. I just want to be able to boot from one of the drives, not from USB/SD card.

Storage management - ssacli tool

To configure the P420i RAID controller, It's highly recommended to install the ssacli tool from HP. Otherwise you have to reboot and wait 10 minutes (no joke) to enter the 'Smart Array Configuration Utility' to make changes to the storage configuration.

I just followed these instructions to install this tool on Ubuntu:

curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add -
curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -

Contents of /etc/apt/sources.list.d/hp.list:

deb http://downloads.linux.hpe.com/SDR/repo/mcp/Ubuntu bionic/current non-free

For Debian:

deb http://downloads.linux.hpe.com/SDR/repo/mcp/debian bookworm/current non-free

Installing the software:

apt-get update
apt-get install ssacli

Some examples:

ssacli ctrl all show detail
ssacli ctrl slot=0 array all show detail
ssacli ctrl slot=0 ld all show detail
ssacli ctrl slot=0 pd all show detail
ssacli ctrl slot=0 ld 7 delete
ssacli ctrl slot=0 create type=ld raid=0 drives=2I:2:7 ss=8 ssdopo=off
ssacli ctrl slot=0 array i modify ssdsmartpath=disable
ssacli ctrl slot=0 pd 1 modify led=on

RAID vs non-RAID

You are forced by the RAID controller to always create a RAID array, even if you don't need the redundancy or performance.

I've chosen to just put SSDs in individual RAID0 logical drives, making them effectively individual drives from the perspective of the operating system. I'm not running a mission-critical application so this is fine for me.

A small downside is that the bright red square light on each drive will permanently glow up. That light signals truthfully that you cannot remove a drive without killing the 'array'. I've not seen any option to turn this off, which I can understand.

Sound / Noise level

Idle sound level

I've taken some sound measurements using my iPhone with an app, so how reliable this is, I don't know but it's something.

During boot time (full fan speed)  : 62 dB
Idle, booted into operating system : 50 dB

To set some expectations:

My very subjective opinion is that at 50 dB the sound level is reasonable for a server like this, but it's definitely not quiet. I would not be able to work, relax or sleep with this server in the same room.

Although this server is fairly quiet at idle, it does need its own dedicated room. When the door is closed, you won't hear it at idle, but under sustained load, you will hear high pitched fan noise even with a closed door.

Impact of adding a PCIe card on noise level

When you put any kind of PCIe card in the server, two of its six fans in-line with the PCIe expansion slot will run at 40%+ to cool these cards.

The sound level will become a bit annoying. I've not found any option to disable this behavior. This means that if you need to expand the server with additional network ports or other components, the server really needs a room with a closed door.

Please note that if you want to keep noise levels down but want to upgrade to 10Gbit networking, you could opt to configure the server with Flexlom 2 x 10Gbit instead of the stock Flexlom 4 x 1 Gbit copper. This will give you more bandwidth without the need to add a PCIe card.

At this time, I have no need to add any high-speed networking as I can run the simulations all on the machine itself (at least that's the plan).

Idle power usage

The DL380p has several BIOS settings that trade performance for (idle) power consumption. I have tested the performance and idle power usage for two power profiles:

HP Power Profile: Balanced Power and Performance
Present Power Reading: 76 Watts
Geekbench Single Core score: 2836

HP Power Profile: Static High Performance
Present Power Reading:  100 Watts
Geekbench Single Core score: 3525

Please note: that actual power usage includes +10 Watt for the iLO interface, to the system would use 86 Watt or 110 Watt at the wall outlet.

What we can learn from the above test results is that the high performance setting uses 31% more power at idle and in return you get 20% more single core performance.

The server supports (obviously) dual power supplies but that option only adds cost, increases power usage and gains me nothing. I'm not running a mission-critical business with this server.

By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall.

Boot up time

I clocked 4 minutes and 28 seconds until I got a ping reply from the operating system.

KVM / iLO (Remote Management)

As with almost all HP servers, this machine has a dedicated remote management port + engine (iLO) so you can do everything remotely, such as powering the server on/off. You also have access to a virtual console (KVM over IP) and virtual CD-ROM drive to remotely provision the server.

For those of you who are less familiar with this hardware: notice the network interface left of the VGA connector. This is the iLO remote management interface. It's a mini computer with its own operatingsystem and IP-address, thus reachable from the network.

In the past iLO or similar solutions were a pain to use because you need Java or .NET to use the virtual console. With the ilO 2.7 update you will have a HTML5 interace, doing away with the need for both Java as .NET. This is a huge usability improvement.

Operating system

I'm currently running Ubuntu Linux 18.04 and using plain vanilla KVM to spawn virtual machines, wich works fine. Everything is maintained/configured with Ansible.

Closing words

I hope this overview was informative. Maybe it could be an option for yourself to consider if you ever want to setup a home lab yourself.

As far as I know this is not 100% reliable as the motherboard serial number can be changed to match the chassis when it's replaced due to failure. I guess it will have to do. ↩

My Ceph test cluster based on Raspberry Pi's and HP MicroServers

2019-01-27T00:00:00+01:00

Introduction

To learn more about Ceph, I've build myself a Ceph Cluster based on actual hardware. In this blogpost I'll discus the cluster in more detail and I've also included (fio) benchmark results.

This is my test Ceph cluster:

The cluster consists of the following components:

 3 x Raspberry Pi 3 Model B+ as Ceph monitors
 4 x HP MicroServer as OSD nodes (3 x Gen8 + 1 x Gen10)
 4 x 4 x 1 TB drives for storage (16 TB raw)
 3 x 1 x 250 GB SSD (750 GB raw)
 2 x 5-port Netgear switches for Ceph backend network (bonding)

Monitors: Raspberry Pi 3 Model B+

I've done some work getting Ceph compiled on a Raspberry Pi 3 Model B+ running Raspbian. I'm using three Raspberry Pi's as Ceph monitor nodes. The Pi boards don't break a sweat with this small cluster setup.

Note: Raspberry Pi's are not an ideal choice as a monitor node because Ceph Monitors write data (probably the cluster state) to disk every few seconds. This will wear out the SD card eventually.

Storage nodes: HP MicroServer

The storage nodes are based on four HP MicroServers. I really like these small boxes, they are sturdy, contain server-grade components, including ECC-memory and have room for four internal 3.5" hard drives. You can also install 2.5" hard drives or SSDs.

For more info on the Gen8 and the Gen10 click on their links.

Unfortunately the Gen8 servers are no longer made. The replacement, the Gen10 model, lacks IPMI/iLO and is also much more expensive (in Europe at least).

CPU and RAM

All HP Microservers have a dual-core CPU. The Gen8 servers have 10GB RAM and the Gen10 server has 12GB RAM. I've just added an 8GB ECC memory module to each server, the Gen10 comes with 4GB and the Gen8 came with only 2GB, which explains the difference.

Boot drive

The systems all have an (old) internal 2.5" laptop HDD connected to the internal USB 2.0 header using an USB enclosure.

Ceph OSD HDD

All servers are fitted with four (old) 1TB 7200 RPM 3.5" hard drives, so the entire cluster contains 16 x 1TB drives.

Ceph OSD SSD

There is a fifth SATA connector on the motherboard, meant for an optional optical drive, which I have no use for and wich is not included with the servers.

I use this SATA connector in the Gen8 MicroServers to attach a Crucial 250GB SSD, which is then tucked away at the top, where the optical drive would sit. So the Gen8 servers have an SSD installed which the Gen10 is lacking.

The entire cluster thus has 3 x 250GB SSDs installed.

Networking

All servers have two 1Gbit network cards on-board and a third one installed in one of the half-height PCIe slots¹.

The half-height PCIe NICs connect the Microservers to the public network. The internal gigabit NICs are configured in a bond (round-robin) and connected to two 5-port Netgear gigabit switches. This is the cluster network or the backend network Ceph uses for replicating data between the storage nodes.

You may notice that the first onboard NIC of each server is connected to the top switch and the second one is connected to the bottom switch. This is necessary because linux round-robin bonding requires either separate VLANs for each NIC or in this case separate switches.

Benchmarks

Benchmark conditions

The tests ran on a physical Ceph client based on an older dual-core CPU and 8GB of RAM. This machine was connected to the cluster with a single gigabit network card.
I've mapped RBD block devices from the HDD pool and the SSD pool on this machine for benchmarking.
All tests have been performed on the raw /dev/rbd0 device, not on any file or filesystem.
The pools use replication with a minimal copy count of 1 and a maximum of 3.
All benchmarks have been performed with FIO.
All benchmarks used random 4K reads/writes

    NAME     ID     USED        %USED     MAX AVAIL     OBJECTS
    hdd      36     1.47TiB     22.64       5.03TiB      396434
    ssd      38      200GiB     90.92       20.0GiB       51204

Benchmark SSD

Click on the images below to see a larger version.

Benchmark HDD

Benchmark evaluation

The random read performance of the hard drives seems unrealistic at higher queue depths and number of simultaneous jobs. This performance cannot be sustained purely on the basis that 16 hard drives with maybe 70 random IOPs each can only sustain 1120 random IOPs.

I cannot explain why I get these numbers. If anybody has a suggestion, feel free to comment/respond. Maybe the total of 42GB of memory across the cluster may act as some kind of cache.

Another interesting observation is that a low number of threads and a small IO queue depth results in fairly poor performance, both for SSD and HDD media.

Especially the performance of the SSD pool is poor with a low IO queue depth. A probable cause is that these SSDs are consumer-grade and don't perform well with low queue depth workloads.

I find it interesting that even over a single 1Gbit link, the SSD-backed pool is able to sustain 20K+ IOPs at higher queue depths and larger number of threads.

The small number of storage nodes and the low number of OSDs per node doesn't make this setup ideal but it does seem to perform fairly decent, considering the hardware involved.

You may notice that the Pi's are missing in the picture because this is an older picture when I was running the monitors as virtual machines on hardware not seen in the picture. ↩

Compiling Ceph on the Raspberry Pi 3B+ (armhf) using Clang/LLVM

2018-11-10T04:00:00+01:00

UPDATE 2019 / 2020

There are official ARM64 binaries of Ceph that you can run on a 64-bit version of Ubuntu 18.04.

Important: I consider this page obsolete. I will keep it up for transparency's sake

Introduction

In this blog post I'll show you how to compile Ceph Luminous for the Raspberry Pi 3B+.

If you follow the instructions below you can compile Ceph on Raspbian. A note of warning: we will compile Ceph on the Raspberry Pi itself which takes a lot of time.

Ubuntu has packages for Ceph on armhf but I was never able to get Ubuntu working properly on the Raspberry Pi 3B+. Maybe that's just me and I did something wrong. Using existing Ceph packages on Ubuntu would probably be the fastest way to get up and running on the Raspberry Pi if it works for you.

This is my test Ceph cluster:

 3 x Raspberry Pi 3B+ as Ceph monitors. 
 4 x HP Microserver as OSD nodes.
 4 x 4 x 1 TB drives for storage (16 TB raw)
 3 x 1 x 250 GB SSD (750 GB raw)
 2 x 5-port Netgear switches for Ceph backend network (bonding)

For the impatient

If you just want the packages you can download this file and you'll get a set of .deb files which you need to install on your Raspberry Pi.

SECURITY WARNING: these packages are created by me, an unknown, untrusted person on the internet. As a general rule you should not download and install these packages as they could be malicious for all you know. If you want to be safe, compile Ceph yourself.

Skip to the section about installing the packages at the end for further installation instructions.

The problem with compiling Ceph for armhf

There are no armhf packages for Ceph because if you try to compile Ceph on armhf the compiler (gcc) will run out of virtual memory (about three gigabytes).

The solution

Daniel Glaser discovered that he could actually compile Ceph on armhf by using Clang/LLVM as the C++ compiler. This compiler seems to use less memory and thus stay within the 3 GB memory boundary. This is why he and I were able to compile Ceph.

How to compile Ceph for armhf - preparation

The challenge: one gigabyte of memory

The Raspberry Pi 3B+ has only one gigabyte of memory but we need more. The only way to add memory is to use swap on disk, as far as I know.

If you use storage as a substitute for RAM memory, you need fast storage, so it's really recommended to use an external SSD drive that you connect through USB. You also may need sufficient storage, I'd recommend 20+ GB.

SD memory cards are not up to the task regarding being used as swap. You'll wear them out prematurely and performance is abysmal. You should really use an external SSD.

Preparing the external SSD

Attach the SSD drive to the Raspberry Pi with USB

The SSD will probably show up as '/dev/sda'.
mkfs.xfs /dev/sda -f ( this will erase all contents of the SSD ).
mkdir /mnt/ssd
mount /dev/sda /mnt/ssd

Creating and activating swap

cd /mnt/ssd
dd if=/dev/zero of=swap.dd bs=1M count=5000
swapon /mnt/ssd/swap.dd
swapoff /var/swap

By default, Raspbian configures a 100 MB swap file on /var/swap. In order to increase performance and protect the SD card from wearing out, please don't forget this last step to disable this swap file on the SD card.

Extra software

I would recommend installing 'htop' for real-time monitoring of cpu, memory and swap usage if you like to do so.

apt-get install htop

How to compile Ceph for armhf - building

Installing an alternative C++ compiler (Clang/LLVM)

As part of Daniel's instructions, you need to compile and install Clang/LLVM. I followed his instructions to the letter, I have not tested the Clang/LLVM packages made available through apt.

Compiling Clang/LLVM takes a lot of time. It took 8 hours to compile LLVM/Clang on a Raspberry Pi 3B+ with make -j3 to limit memory usage.

real    493m38.472s
user    1223m39.063s
sys 45m45.748s

I'll reproduce the steps from Daniel here:

apt update
apt install -y build-essential ca-certificates vim git 
apt install libcunit1-dev libcurl4-openssl-dev python-bcrypt python-tox python-coverage

cd /mnt/ssd
mkdir git && cd git
git clone https://github.com/llvm-mirror/llvm.git
cd llvm/tools
git clone https://github.com/llvm-mirror/clang.git
git clone https://github.com/llvm-mirror/lld.git
cd /tmp
mkdir llvm-build && cd llvm-build
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=ARM /mnt/ssd/git/llvm/
make -j3
make install
update-alternatives --install /usr/bin/cc cc /usr/local/bin/clang 100
update-alternatives --install /usr/bin/c++ c++ /usr/local/bin/clang++ 100
update-alternatives --install /usr/bin/cpp cpp /usr/local/bin/clang-cpp 100

You may chose to build in some other directory, maybe on the SSD itself. I'm not sure if that makes a big difference. Be carefull when using /tmp as all contents are lost after a reboot.

Obtaining Ceph

There are two options: 1. clone my Luminous fork containing the branch 'ceph-on-arm' which incorporates all the 'fixed' files that make Ceph build with Clang/LLVM.

Clone the official Ceph repo and use the luminous branche. Next, you edit all the relevant files and make the changes yourself. Here you can find a list of all the files and the changes made to them.

I would recommend to just git clone ceph like this:

cd /mnt/ssd
git clone https://github.com/louwrentius/ceph
cd ceph
git checkout ceph-on-arm
git reset --hard
git clean -dxf
git submodule update --init --recursive

Now we first need to install a lot of dependancies on the Raspberry Pi before we can build Ceph.

run ./install-deps.sh

This will take some time as a ton of packages will be installed. Once this is done we are ready to compile Ceph itself.

Building Ceph

So to understand what you are getting into: it took me about 12 hours to compile Ceph on a Raspberry Pi 3 B+

real    717m31.457s
user    1319m50.438s
sys 58m7.549s

This is the command to run:

./make-debs.sh

If you want to monitor cpu and memory usage, you can use 'htop' to do so.

If for some reason the compile proces does fail and you may have to restart compiling ceph after you made some adjustments:

(you may have to adjust the folder name to match your ceph version)

cd /tmp/release/Raspbian/WORKDIR/ceph-12.2.9-39-gd51dfb14f4
< edit the relevant files here >
dpkg-buildpackage -j3 -us -us -nc

Once this process is done, you will find a lot of .deb packages in your /tmp/release/Raspbian/WORKDIR folder.

Warning If you do use /tmp, the first thing to do is to copy all .deb files to a safe location because if you reboot your Pi, you loose 12 hours of work.

Assuming that you copied all .deb files to a folder like '/deb' you just created, this is how you install these packages:

dpkg --install *.deb
apt-get install --fix-missing
apt --fix-broken install

This is a bit ugly but it worked fine for me.

You can now just copy over all the .deb files to other Raspbery Pi's and install Ceph on them too.

Now you are done and you can run Ceph on a Raspberry Pi 3B+.

Ceph monitors may wear out the SD card

Important Running a Ceph monitor node on a Raspberry Pi is not ideal. The core issue is that the Ceph monitor process writes data every few seconds to files within /var/lib/ceph and this may wear out the SD card prematurely. The solution would be to use an external usb hard drive mounted through USB or a regular ssd which is way more resilient to writes than a regular SD card.

Understanding Ceph: open-source scalable storage

2018-08-19T04:00:00+02:00

Introduction

In this blog post I will try to explain why I believe Ceph is such an interesting storage solution. After you finished reading this blog post you should have a good high-level overview of Ceph.

I've written this blog post purely because I'm a storage enthusiast and I find Ceph interesting technology.

What is Ceph?

Ceph is a software-defined storage solution that can scale both in performance and capacity. Ceph is used to build multi-petabyte storage clusters.

For example, Cern has build a 65 Petabyte Ceph storage cluster. I hope that number grabs your attention. I think it's amazing.

The basic building block of a Ceph storage cluster is the storage node. These storage nodes are just commodity (COTS) servers containing a lot of hard drives and/or flash storage.

Example of a storage node

Ceph is meant to scale. And you scale by adding additional storage nodes. You will need multiple servers to satisfy your capacity, performance and resiliency requirements. And as you expand the cluster with extra storage nodes, capacity, performance and resiliency (if needed) will all increase at the same time.

It's that simple.

You don't need to start with petabytes of storage. You can actually start very small, with just a few storage nodes and expand as your needs increase.

I want to touch upon a technical detail because it illustrates the mindset surrounding Ceph. With Ceph, you don't even need a RAID controller anymore, a 'dumb' HBA is sufficient. This is possible because Ceph manages redundancy in software. A Ceph storage node at it's core is more like a JBOD. The hardware is simple and 'dumb', the intelligence resides all in software.

This means that the risk of hardware vendor lock-in is quite mitigated. You are not tied to any particular proprietary hardware.

What makes Ceph special?

At the heart of the Ceph storage cluster is the CRUSH algoritm, developed by Sage Weil, the co-creator of Ceph.

The CRUSH algoritm allows storage clients to calculate which storage node needs to be contacted for retrieving or storing data. The storage client can - on it's own - determine what to do with data or where to get it.

So to reiterate: given a particular state of the storage cluster, the client can calculate which storage node to contact for storage or retrieval of data.

Why is this so special?

Because there is no centralised 'registry' that keeps track of the location of data on the cluster (metadata). Such a centralised registry can become:

a performance bottleneck, preventing further expansion
a single-point-of-failure

Ceph does away with this concept of a centralised registry for data storage and retrieval. This is why Ceph can scale in capacity and performance while assuring availability.

At the core of the CRUSH algoritm is the CRUSH map. That map contains information about the storage nodes in the cluster. That map is the basis for the calculations the storage client need to perform in order to decide which storage node to contact.

This CRUSH map is distributed across the cluster from a special server: the 'monitor' node. Regardless of the size of the Ceph storage cluster, you typically need just three (3) monitor nodes for the whole cluster. Those nodes are contacted by both the storage nodes and the storage clients.

So Ceph does have some kind of centralised 'registry' but it serves a totally different purpose. It only keeps track of the state of the cluster, a task that is way easier to scale than running a 'registry' for data storage/retrieval itself.

It's important to keep in mind that the Ceph monitor node does not store or process any metadata. It only keeps track of the CRUSH map for both clients and individual storage nodes. Data always flows directly from the storage node towards the client and vice versa.

Ceph Scalability

A storage client will contact the appropriate storage node directly to store or retrieve data. There are no components in between, except for the network, which you will need to size accordingly¹.

Because there are no intermediate components or proxies that could potentially create a bottleneck, a Ceph cluster can really scale horizontally in both capacity and performance.

And while scaling storage and performance, data is protected by redundancy.

Ceph redundancy

Replication

In a nutshell, Ceph does 'network' RAID-1 (replication) or 'network' RAID-5/6 (erasure encoding). What do I mean by this? Imagine a RAID array but now also imagine that instead of the array consisting of hard drives, it consist of entire servers.

That's what Ceph does: it distributes the data across multiple storage nodes and assures that the copy of a piece of data is never stored on the same storage node.

This is what happens if a client writes two blocks of data:

Notice how a copy of the data block is always replicated to other hardware.

Ceph goes beyond the capabilities of regular RAID. You can configure more than one replica. You are not confined to RAID-1 with just one backup copy of your data². The only downside of storing more replicas is the storage cost.

You may decide that data availability is so important that you may have to sacrifice space and absorb the cost. Because at scale, a simple RAID-1 replication scheme may not sufficiently cover the risk and impact of hardware failure anymore. What if two storage nodes in the cluster die?

This example or consideration has nothing to do with Ceph, it's a reality you face when you operate at scale.

RAID-1 or the Ceph equivalent 'replication' offers the best overall performance but as with 'regular' RAID-1, it is not very storage space efficient. Especially if you need more than one replica of the data to achieve the level of redundancy you need.

This is why we used RAID-5 and RAID-6 in the past as an alternative to RAID-1 or RAID-10. Parity RAID assures redundancy but with much less storage overhead at the cost of storage performance (mostly write performance). Ceph uses 'erasure encoding' to achieve a similar result.

Erasure Encoding

With Ceph you are not confined to the limits of RAID-5/RAID-6 with just one or two 'redundant disks' (in Ceph's case storage nodes). Ceph allows you to use Erasure Encoding, a technique that let's you tell Ceph this:

"I want you to chop up my data in 8 data segments and 4 parity segments"

These segments are then scattered across the storage nodes and this allows you to lose up to four entire hosts before you hit trouble. You will have only 33% storage overhead for redundancy instead of 50% (or even more) you may face using replication, depending on how many copies you want.

This example does assume that you have at least 8 + 4 = 12 storage nodes. But any scheme will do, you could do 6 data segments + 2 parity segments (similar to RAID-6) with only 8 hosts. I think you catch the idea.

Ceph failure domains

Ceph is datacenter-aware. What do I mean by that? Well, the CRUSH map can represent your physical datacenter topology, consisting of racks, rows, rooms, floors, datacenters and so on. You can fully customise your topology.

This allows you to create very clear data storage policies that Ceph will use to assure the cluster can tollerate failures across certain boundaries.

An example of a topology:

If you want, you can lose a whole rack. Or a whole row of racks and the cluster could still be fully operational, although with reduced performance and capacity.

That much redundancy may cost so much storage that you may not want to employ it for all of your data. That's no problem. You can create multiple storage pools that each have their own protection level and thus cost.

How do you use Ceph?

Ceph at it's core is an object storage solution. Librados is the library you can include within your software project to access Ceph storage natively. There are Librados implementations for the following programming languages:

C(++)
Java
Python
PHP

Many people are looking for more traditional storage solutions, like block storage for storing virtual machines, a POSIX compliant shared file system or S3/OpenStack Swift compatible object storage.

Ceph provides all those features in addition to it's native object storage format.

I myself are mostly interested in block storage (Rados Block Device)(RBD) with the purpose of storing virtual machines. As Linux has native support for RBD, it makes total sense to use Ceph as a storage backend for OpenStack or plain KVM.

With very recent versions of Ceph, native support for iSCSI has been added to expose block storage to non-native clients like VMware or Windows. For the record, I have no personal experience with this feature (yet).

The Object Storage Daemon (OSD)

In this section we zoom in a little bit more into the technical details of Ceph.

If you read about Ceph, you read a lot about the OSD or object storage daemon. This is a service (daemon) that runs on the storage node. The OSD is the actual workhorse of Ceph, it serves the data from the hard drive or ingests it and stores it on the drive. The OSD also assures storage redunancy, by replicating data to other OSDs based on the CRUSH map.

To be precise: for every hard drive or solid state drive in the storage node, an OSD will be active. Does your storage node have 24 hard drives? Then it runs 24 OSDs.

And when a drive goes down, the OSD will go down too and the monitor nodes will redistribute an update CRUSH map so the clients are aware and know where to get the data. The OSDs also respond to this update, because redundancy is lost, they may start to replicate non-redundant data to make it redundant again (across fewer nodes).

When the drive is replaced, the cluster will 'self-heal'. This means that the new drive will be filled with data once again to make sure data is spread evenly across all drives within the cluster.

So maybe it's interesting to realise that storage clients effectively directly talk to the OSDs that in turn talk to the individual hard drives. There aren't many components between the client and the data itself.

Closing words

I hope that this blog post has helped you understand how Ceph works and why it is so interesting. If you have any questions or feedback please feel free to comment or email me.

If you have a ton of high-volume sequential data storage traffic, you should realise that a single host with a ton of drives can easily saturate 10Gbit or theoretically even 40Gbit. I'm assuming 150 MB/s per hard drive. With 36 hard drives you would face 5.4 GB/s. Even if you only would run half that speed, you would need to bond multiple 10Gbit interfaces to sustain this load. Imagine the requirements for your core network. But it really depends on your workload. You will never reach this kind of throughput with a ton of random I/O unless you are using SSDs, for instance. ↩
Please note that in production setups, it's the default to have a total of 3 instances of a data block. So that means 'the original' plus two extra copies. See also this link. Thanks to sep76 from Reddit to point out that the default is 3 instances of your data. ↩

Setup a VPN on your iPhone with OpenVPN and Linux

2018-06-18T04:00:00+02:00

[Update 2018] This article has been substantially updated since it was published in 2013.

Introduction

In this article, I will show you how to setup a Linux-based OpenVPN server. Once this server is up and running, I'll show you how to setup your iOS devices, such as your iPhone or iPad so that they can connect with your new VPN server.

The goal of this effort is to encapsulate all internet traffic through your VPN connection so no matter where you are, nobody can monitor which sites you visit and what you do. This is ideal if you have to visit the internet through untrusted internet sources like public Wi-Fi.

Some typical scenarios would be:

you run an OpenVPN service on your Linux-based home router directly
you run an OpenVPN service on a device behind your home router using portforwarding (like a Raspberry Pi)
you run an OpenVPN service on a VPS hosted by one of many cloud service providers

Your iOS devices will be running OpenVPN Connect, a free application found in the App store.

A note on other platforms: Although this tutorial is focussed on iOS devices, your new OpenVPN-based VPN server will support any client OS, may it be Windows, MacOS, Android or Linux. Configuration of these other clients is out-of-scope for this article.

This tutorial is based on OpenVPN, an open-source product. The company behind OpenVPN also offers VPN services for a price per month. If you find the effort of setting up your own server too much of a hassle, you could look into their service. Please note that I have never used this service and cannot vouch for it.

This is a brief overview of all the steps you will need to take in order to have a fully functional setup, including configuration of the clients:

Install a Linux server (out-of-scope)
Install the OpenVPN software
Setup the Certificate Authority
Generate the server certificate
Configure the OpenVPN server configuration
Configure the firewall on your Linux server
Generate certificates for every client (iPhone, iPad, and so on)
Copy the client configuration to your devices
Test your clients

How It Works

OpenVPN is an SSL-based VPN solution. SSL-based VPNs are very reliable because if you set it up properly, you will never be blocked by any firewall as long as TCP-port 443 is accessible. By default, OpenVPN uses UDP as a transport at port 1194, but you can switch to TCP-port 443 to increase the chance that your traffic will not be blocked at the cost of a little bit more bandwidth usage.

Authentication

Authentication is based on public/private key cryptography. The OpenVPN server is similar to an HTTPS server. The biggest difference is that your device doesn't use a username/password combination for authentication, but a certificate. This certificate is stored within the client configuration file.

So before you can configure and start your OpenVPN service, you need to setup a Certificate Authority (CA). With the CA you can create the server certificate for your OpenVPN server and after that's done, generate all client certificates.

OpenVPN installation

OpenVPN is available on most common Linux Distros by default. apt-get install openvpn for any Debian or Ubuntu version is all you need to install OpenVPN.

Or take a look here

I have never tried it out, but you can try and take a look at an OpenVPN install script

This script seems to automate a lot of steps, like firewall configuration, certificate generation, etc.

Tip

It's out-of-scope for this tutorial, but you should make sure that you keep your OpenVPN software up-to-date, in case security vulnerabilities are discovered in OpenVPN in the future.

Security

I'm creating this tutorial on an older system, with less secure default configuration settings for both the Certificate Authority as the OpenVPN server itself. The settings I use in this tutorial are based on the steps in this blog.

Notable improvements:

AES256 for encryption
2048 bit key sizes over 1024 bit keys
SHA256 over sha1/md5

Performance

I did some performance tests and got around 40-50 Mbs per iOS client. I believe that the bottleneck lies with my old HP Microserver N40L with its relatively weak CPU.

Traffic Shaping

If you want to limit how much bandwidth a client is allowed to use, I recommend to use this tutorial. I have tried it out and it works perfectly.

Creating a certificate authority.

For unbuntu: install the package "easy-rsa" and use the 'make-cadir' command instead of the setup instructions below.

I assume that you will setup your OpenVPN configuration in /etc/openvpn. Before you can setup the server configuration, you need to create a certificate authority. I used the folder /etc/openvpn/easy-rsa as the location for my CA.

mkdir /etc/openvpn/easy-rsa

We start with copying all these files to this new directory:

cp -R /usr/share/doc/openvpn/examples/easy-rsa/2.0* /etc/openvpn/easy-rsa

Please note that depending on your Linux flavour, these files may be found at some other path.

Next, we cd into the destination directory.

cd /etc/openvpn/easy-rsa

Now, open the 'vars' file with your favorite text editor. The following instructions are straight from the OpenVPN howto.

You should change all the values to ones that apply to you (obviously).

export KEY_COUNTRY="US"
export KEY_PROVINCE="California"
export KEY_CITY="San Fransisco"
export KEY_ORG="My Company"
export KEY_EMAIL="my@mail.com"
export KEY_CN=server
export KEY_NAME=server
export KEY_OU=home

Change the KEY_SIZE parameter:

export KEY_SIZE=2048

How long would you like your certificates to be valid (10 years?)

export CA_EXPIRE=3650
export KEY_EXPIRE=3650

Then I had to copy openssl-1.0.0.cnf to openssl.cnf because the 'vars' script complained that it couldn't find the latter file.

cp openssl-1.0.0.cnf openssl.cnf

Notice I went through these steps on an older Linux installation. I had to edit the file /etc/openvpn/easy-rsa/pkitool and changed all occurrences of 'sha1' to 'sha256'.

Now we 'source' var and run two additional commands that actually generate the certificate authority. Notice the dot before ./vars.

. ./vars
./clean-all
./build-ca
./build-dh

You will have to confirm the values or change them if necessary.

Now we have a certificate authority and we can create new certificates that will be signed by this authority.

WARNING: be extremely careful with all key files, they should be kept private.

I would recommend performing these commands:

chown -R root:root /etc/openvpn 
chmod -R 700 /etc/openvpn

By default, OpenVPN runs as root. With these commands, only the root user will be able to access the keys. If you don't run OpenVPN as root, you must select the appropriate user for the first command. See also this article.

Creating the Server Certificate

We create the server certificate:

./build-key-server server

It's up to you to come up with an alternative for 'server'. This is the file name under which the key files and certificates are stored.

All files that are generated can be found in the '/etc/openvpn/easy-rsa/keys' directory. This is just a flat folder with both the server and client keys.

Creating the optional TLS-AUTH Certificate

This step is optional but it doesn't take much effort and it seems to add an additional security layer at no significant cost. In this step we create an additional secret key that is shared with both the server and the clients.

The following steps are based on this article (use of -tls-auth).

cd /etc/openvpn/easy-rsa/keys
openvpn --genkey --secret ta.key

When we are going to create the server configuration, we will reference this key file.

Creating the Client Certificate

Now that we have a server certificate, we are going to create a certificate for our iPhone (or any other iOS device).

./build-key iphone

Answer the questions with the defaults. Don't forget to answer these questions:

Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y

So now we have everything in place to start creating an OpenVPN configuration. We must create a configuration for the server and the client. Those configurations are based on the examples that can be find in /usr/share/doc/openvpn/examples/.

Example Server configuration

This is my server configuration which is operational. It is stored in /etc/openvpn/openvpn.conf

dev tun2
tls-server
cipher AES-256-CBC
auth SHA256
remote-cert-tls client
dh easy-rsa/keys/dh2048pem
ca easy-rsa/keys/ca.crt
cert easy-rsa/keys/server.crt
key easy-rsa/keys/server.key
tls-auth easy-rsa/keys/ta.key
server 10.0.0.0 255.255.255.0
log /var/log/openvpn.log
script-security 2
route-up "/sbin/ifconfig tun2 up"
port 443
proto tcp-server
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"

I believe you should be able to use this configuration as-is. Depending on your local IP-addresses within your own network, you may have to change the server section.

I use TCP-port 443 as this destination port is almost never blocked as blocking this port would break most internet connectivity. (The downside is that I can no longer host any secure web site on this IP-address).

The OpenVPN service will provide your client with an IP-address within the address range configured in the 'server' section.

Change any parameters if required and then start or restart the OpenVPN service:

/etc/init.d/openvpn restart

Make sure that the server is running properly in /var/log/openvpn.log

If you want to use your VPN to browse the internet, we still need to configure a basic firewall setup.

I'm assuming that you already have some kind of IPtables-based firewall running. Configuring a Linux firewall is out-of-scope for this article. I will only discuss the changes you may need to make for the OpenVPN service to operate properly.

You will need to accept traffic to TCP port 443 on the interface connected to the internet.

iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

If your OpenVPN server is behind a router/firewall, you need to configure port-forwarding on that router/firewall. How to do so is out-of-scope for this article, as it is different for different devices.

Assuming that you will - for example - use the 10.0.0.0/24 network for VPN clients such as your iPhone, you must also create a NAT rule so VPN clients can use the IP-address of the Linux server to access Internet.

iptables -t nat -A POSTROUTING -s "10.0.0.0/24" -o "eth0" -j MASQUERADE

Please note that you must change eth0 with the name of the appropriate interface that connects to the internet. Change the IP-address range according to your own situation. It should not conflict with your existing network.

iptables -A FORWARD -p tcp -s 10.0.0.0/24 -d 0.0.0.0/0 -j ACCEPT

Please note that I haven't tested these rules, as I have a different setup. But this should be sufficient. And make sure that forwarding is enabled like this:

echo 1 > /proc/sys/net/ipv4/ip_forward

Example Client configuration

Most OpenVPN clients can automatically import files with the .ovpn file extension. A typical configuration file is something like 'iphone.ovpn'.

Warning: the .ovpn files will contain the certificate used by your iPhone/iPad to authenticate against your OpenVPN server. Be very carefull where you store this file. Anyone that is able to obtain a copy of this file, will be able to connect to your VPN server.

This is an example configuration file, but we are not going to create it by hand, it's too much work.

What you will notice from this example is that the .ovpn file contains both the client configuration and all the required certificates:

the CA root certificate
the server certificate to validate the server
the client private certificate
the TLS-AUTH certificate (an optional extra security measure)

Create a client configuration file (.ovpn) with a script

You can create your client configuration file manually but that is a lot of work. Because you need to append all the certificates to a single file, that also contains the configuration settings.

So we will use a script to setup the client configuration.

First we are going to create a folder where our client configuration files will be stored.

mkdir /etc/openvpn/clientconfig
chmod 700 /etc/openvpn/clientconfig

Now we will download the script and the accompanying configuration template file. Notice that the links may wrap.

cd /etc/openvpn
wget https://raw.githubusercontent.com/louwrentius/openvpntutorial/master/create-client-config.sh
wget https://raw.githubusercontent.com/louwrentius/openvpntutorial/master/client-config-template
chmod +x create-client-config.sh

Please note that you first need to create the certificates for your devices before you can generate a configuration file. So please go back to that step if you need to.

Also take note of the name you have used for your devices. You can always take a look in /etc/openvpn/easy-rsa/keys to see how your devices are called.

Now, edit this client-config-template and change the appropriate values where required. You may probably only need to change the first line:

remote <your server DNS address or IP address>

Now you are ready to run the script and generate the config file for your device.

When you run this script, a configuration file is generated and placed in to the folder /etc/openvpn/clientconfig.

The script just puts the client configuration template and all required certificates in one file. This is how you use it:

./create-client-config.sh iPhone

Some output you will notice when running the script:

user@server:/etc/openvpn# ./create-client-config.sh iphone
Client's cert found: /etc/openvpn/easy-rsa/keys/iphone
Client's Private Key found: /etc/openvpn/easy-rsa/keys/iphone.key
CA public Key found: /etc/openvpn/easy-rsa/keys/ca.crt
tls-auth Private Key found: /etc/openvpn/easy-rsa/keys/ta.key
Done! /etc/openvpn/clientconfig/iphone.ovpn Successfully Created.

You should now find a file called 'iphone.ovpn' in the directory /etc/openvpn/clientconfig.

We are almost there. We just need to copy this file to your iOS device.

You have three options:

Use iCloud Drive
Use iTunes
Use email (obviously insecure and not discussed)

Setting up your iPhone or iPad with iCloud Drive

First install the OpenVPN Connect application if you haven't done so.
Copy the .ovpn file from your OpenVPN server to your iCloud Drive.
Take your device and use the 'files' browser to navigate within your iCloud drive to the .ovpn file you just copied.
Tap on the file to download and open it.
Now comes the tricky part: press the share symbol

Open the file with the OpenVPN application on your iOS device:

When you get the question "OpenVPN would like to Add VPN Configurations", choose 'Allow'.

Continue with the step 'Test your iOS device'.

If the OpenVPN Connect client doesn't import the file, remove the application from the device and re-install it. (This is what I had to do on my iPad).

Setting up your iPhone or iPad with iTunes

You can skip this step if you used iCloud Drive to copy the .ovpn profile to your device.

You need to get the following files on your iOS device:

iphone.ovpn

Copy this file from your OpenVPN server to the computer running iTunes. Then connect your device to iTunes with a cable.

Open iTunes
Select your device at the top right
Go to the Apps tab
Scroll to the file sharing section
Select the OpenVPN application
Add the iphone.ovpn
Sync your device

Test your iOS device

Open the OpenVPN client. You will see a notice that a new configuration has been imported and you need to accept this configuration.

As it might not work straight away, you need to monitor /var/log/openvpn.log on the server to watch for any errors.

Now try to connect and enjoy.

Conclusion

You should be able to keep your VPN enabled at all times because battery usage overhead should be minimal. If you are unable to connect to your VPN when you are at home behind your own firewall, you need to check your firewall settings.

Updated 20130123 with keepalive option. Updated 20130801 with extra server push options for traffic redirection and DNS configuration Updated 20180618 as substantial rewrite of the original outdated article.

HP Proliant Microserver Gen10 as router or NAS

2017-09-14T12:00:00+02:00

Introduction

In the summer of 2017, HP released the Proliant Microserver Gen10. This machine replaces the older Gen8 model.

For hobbyists, the Microserver always has been an interesting device for a custom home NAS build or as a router.

Let's find out if this is still the case.

Price

In The Netherlands, the price of the entry-level model is similar to the Gen8: around €220 including taxes.

CPU

The new AMD X3216 processor has slightly better single threaded performance as compared to the older G1610t in the Gen8. Overall, both devices seem to have similar CPU performance.

The biggest difference is the TDP: 35 Watt for the Celeron vs 15 Watt for the AMD CPU.

Memory

By default, it has 8 GB of unbuffered ECC memory, that's 4 GB more than the old model. Only one of the two memory slots is occupied, so you can double that amount just by adding another 8 GB stick. It seems that 32 GB is the maximum.

Storage

This machine has retained the four 3.5" drive slots. There are no drive brackets anymore. Before inserting a hard drive, you need to remove a bunch of screws from the front of the chassis and put four of them in the mounting holes of each drive. These screws then guide the drive through grooves into the drive slot. This caddy-less design works perfectly and the drive is mounted rock-solid in it's position.

To pop a drive out, you have to press the appropriate blue lever, which latches on to one of the front screws mounted on your drive and pulls it out of the slot.

There are two on-board sata controllers.

00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 49)
01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb/s Controller (rev 11)

The Marvell controller is connected to the four drive bays. The AMD controller is probably connected to the fifth on-board SATA port.

As with the Gen8, you need a floppy-power-connector-to-sata-power-connector cable if you want to use a SATA drive with the fifth onboard SATA port.

Due to the internal SATA header or the USB2.0 header, you could decide to run the OS without redundancy and use all four drive bays for storage. As solid state drives tend to be very reliable, you may use a small SSD to keep the cost and power usage down and still retain reliability (although not the level of reliability RAID1 provides).

Networking

Just as the Gen8, the Gen10 has two Gigabit network cards. The brand and model is: Broadcom Limited NetXtreme BCM5720

As tested with iperf3 I get full 1 Gbit network performance. No problems here (tested on CentOS 7).

PCIe slots

This model has two half-height PCIe slots (1x and 8x in a 4x and 8x physical slot) which is an improvement over the single PCIe slot in the Gen8.

USB

The USB configuration is similar to the Gen8, with both USB2 and USB3 ports and one internal USB2 header on the motherboard.

Sidenote: the onboard micro SD card slot as found in the Gen8 is not present in the Gen10.

Graphics

The Gen10 has also a GPU build-in but I have not looked into it as I have no use for it.

The Gen10 differs in output options as compared to the Gen8: it supports one VGA and two displayport connections. Those displayport connectors could make the Gen10 an interesting DIY HTPC build, but I have not looked into it.

iLO

The Gen10 has no support for iLO. So no remote management, unless you have an external KVM-over-IP solution.

This is a downside, but for home users, this is probably not a big deal. My old Microserver N40L didn't have iLO and it never bothered me.

And most of all: iLO is a small on-board mini-comuter that increases idle power consumption. So the lack of iLO support should mean better idle power consumption.

Boot

Both Legacy and UEFI boot is supported. I have not tried UEFI booting.

Booting from the 5th internal SATA header is supported and works fine (as opposed to the Gen8).

For those who care: booting is a lot quicker as opposed to the Gen8, which took ages to get through the BIOS.

Power Usage

I have updated this segment as I have used some incorrect information in the original article.

The Gen10 seems to consume 14 Watt at idle, booted into Centos 7 without any disk drives attached (removed all drives after booting). This 14 Watt figure is reported by my external power meter.

Adding a single old 7200 1 TB drive drives power usage up to 21 Watt (as expected).

With four older 7200 RPM drives the entire system uses about 43 Watt according to the external power meter.

As an experiment, I've put two old 60 GB 2.5" laptop drives in the first two slots, configured as RAID1. Then I added two 1 TB 7200 RPM drives to fill up the remaining slots. This resulted in a power usage of 32 Watt.

Dimensions and exterior

Exactly the same as the Gen8, they stack perfectly.

The Gen8 had a front door protecting the drive bays connected to the chassis with two hinges. HP has been cheap on the Gen10, so when you open the door, it basically falls off, there's no hinge. It's not a big issue, the overall build quality of the Gen10 is excellent.

I have no objective measurements of noise levels, but the device seems almost silent to me.

Evaluation and conclusion

At first, I was a bit disappointed about the lack of iLO, but it turned out for the best. What makes the Gen10 so interesting is the idle power consumption. The lack of iLO support probably contributes to the improved idle power consumption.

The Gen8 measures between 30 and 35 Watt idle power consumption, so the Gen10 does fare much better (~18 Watt).

Firewall/Router

At this level of power consumption, the Gen10 could be a formidable router/firewall solution. The only real downside is it's size as compared to purpose-built firewalls/routers. The two network interfaces may provide sufficient network connectivity but if you need more ports and using VLANs is not enough, it's easy to add some extra ports.

If an ancient N40L with a piss-poor Atom processor can handle a 500 Mbit internet connection, this device will have no problems with it, I'd presume. Once I've taken this device into production as a replacement for my existing router/firewall, I will share my experience.

Storage / NAS

The Gen8 and Gen10 both have four SATA drive bays and a fifth internal SATA header. From this perspective, nothing has changed. The reduced idle power consumption could make the Gen10 an even more attractive option for a DIY home grown NAS.

All things considered I think the Gen10 is a great device and I have not really encountered any downsides. If you have no problems putting a bit of effort into a DIY solution, the Gen10 is a great platform for a NAS or Router/Firewall, that can compete with most purpose-build devices.

I may update this article as I gain more experience with this device.

Using InfiniBand for cheap and fast point-to-point Networking

2017-03-25T12:00:00+01:00

InfiniBand networking is quite awesome. It's mainly used for two reasons:

low latency
high bandwidth

As a home user, I'm mainly interested in setting up a high bandwidth link between two servers.

I was using quad-port network cards with Linux Bonding, but this solution has some downsides:

you can only go to 4 Gbit with Linux bonding (or you need more ports)
you need a lot of cabling
it is similar in price as InfiniBand

So I've decided to take a gamble on some InfiniBand gear. You only need InfiniBand PCIe network cards and a cable.

1 x SFF-8470 CX4 cable                                              $16
2 x MELLANOX DUAL-PORT INFINIBAND HOST CHANNEL ADAPTER MHGA28-XTC   $25
                                                            Total:  $66

I find $66 quite cheap for 20 Gbit networking. Regular 10Gbit Ethernet networking is often still more expensive that using older InfiniBand cards.

InfiniBand is similar to Ethernet, you can run your own protocol over it (for lower latency) but you can use IP over InfiniBand. The InfiniBand card will just show up as a regular network device (one per port).

ib0 Link encap:UNSPEC HWaddr 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00  
      inet addr:10.0.2.3  Bcast:10.0.2.255  Mask:255.255.255.0
      inet6 addr: fe80::202:c902:29:8e01/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
      RX packets:7988691 errors:0 dropped:0 overruns:0 frame:0
      TX packets:17853128 errors:0 dropped:10 overruns:0 carrier:0
      collisions:0 txqueuelen:256 
      RX bytes:590717840 (563.3 MiB)  TX bytes:1074521257501 (1000.7 GiB)

Configuration

I've followed these instructions to get IP over InfiniBand working.

Modules

First, you need to assure the following modules are loaded at a minimum:

ib_mthca
ib_ipoib

I only had to add the ib_ipoib module to /etc/modules. As soon as this module is loaded, you will notice you have some ibX interfaces available which can be configured like regular ethernet cards

Subnet manager

In addition to loading the modules, you also need a subnet manager. You just need to install it like this:

apt-get install opensm

This service needs to run on just one of the endpoints.

Link status

if you want you can check the link status of your InfiniBand connection like this:

# ibstat
CA 'mthca0'
    CA type: MT25208
    Number of ports: 2
    Firmware version: 5.3.0
    Hardware version: 20
    Node GUID: 0x0002c90200298e00
    System image GUID: 0x0002c90200298e03
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 20
        Base lid: 1
        LMC: 0
        SM lid: 2
        Capability mask: 0x02510a68
        Port GUID: 0x0002c90200298e01
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02510a68
        Port GUID: 0x0002c90200298e02
        Link layer: InfiniBand

Set mode and MTU

Since my systems run Debian Linux, I've configured /etc/network/interfaces like this:

auto ib0
iface ib0 inet static
    address 10.0.2.2
    netmask 255.255.255.0
    mtu 65520
    pre-up echo connected > /sys/class/net/ib0/mode

Please take note of the 'mode' setting. The 'datagram' mode gave abysmal network performance (< Gigabit). The 'connected' mode made everything perform acceptable.

The MTU setting of 65520 improved performance by another 30 percent.

Performance

I've tested the card on two systems based on the Supermicro X9SCM-F motherboard. Using these systems, I was able to achieve file transfer speeds up to 750 MB (Megabytes) per second or about 6.5 Gbit as measured with iperf.

~# iperf -c 10.0.2.2
------------------------------------------------------------
Client connecting to 10.0.2.2, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 10.0.2.3 port 40098 connected with 10.0.2.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.49 GBytes  6.43 Gbits/sec

Similar test with netcat and dd:

~# dd if=/dev/zero bs=1M count=100000 | nc 10.0.2.2 1234
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 128.882 s, 814 MB/s

Testing was done on Debian Jessie.

During earlier testing, I've also used these cards in HP Micro Proliant G8 servers. On those servers, I was running Ubuntu 16.04 LTS.

As tested on Ubuntu with the HP Microserver:

------------------------------------------------------------
Client connecting to 10.0.4.3, TCP port 5001
TCP window size: 4.00 MByte (default)
------------------------------------------------------------
[  5] local 10.0.4.1 port 52572 connected with 10.0.4.3 port 5001
[  4] local 10.0.4.1 port 5001 connected with 10.0.4.3 port 44124
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.0 sec  71.9 GBytes  10.3 Gbits/sec
[  4]  0.0-60.0 sec  72.2 GBytes  10.3 Gbits/sec

Using these systems, I was able eventually able to achieve 15 Gbit as measured with iperf, although I have no 'console screenshot' from it.

Closing words

IP over InfiniBand seems to be a nice way to get high-performance networking on the cheap. The main downside is that when using IP over IB, CPU usage will be high.

Another thing I have not researched, but could be of interest is running NFS or other protocols directly over InfiniBand using RDMA, so you would bypass the overhead of IP.

Tracking down a faulty Storage Array Controller with ZFS

2016-12-15T12:00:00+01:00

One day, I lost two virtual machines on our DR environment after a storage vMotion.

Further investigation uncovered that any storage vMotion of a virtual machine residing on our DR storage array would corrupt the virtual machine's disks.

I could easily restore the affected virtual machines from backup and once that was done, continued my investigation.

I needed a way to quickly verifying if a virtual hard drive of a virtual machine was corrupted after a storage vMotion to understand what the pattern was.

First, I created a virtual machine based on Linux and installed ZFS. Then, I attached a second disk of about 50 gigabytes and formatted this drive with ZFS. Once I filled the drive using 'dd' to about 40 gigabytes I was ready to test.

ZFS was chosen for testing purposes because it stores hashes of all blocks of data. This makes it very simple to quickly detect any data corruption. If the hash doesn't match the hash generated from the data, you just detected corruption.

Other file systems don't store hashes and don't check for data corruption so they just trust the storage layer. It may take a while before you find out that data is corrupted.

I performed a storage vMotion of this secondary disk towards different datastores and then ran a 'zfs scrub' to track down any corruption. This worked better than expected: the scrub command would hang if the drive was corrupted by the storage vMotion. The test virtual machine required a reboot and a reformat of the secondary hard drive with ZFS as the previous file system, including data got corrupted.

After performing a storage vMotion on the drive in different directions, from different datastores to other datastores slowly a pattern emerged.

Storage vMotion corruption happened independent of the VMware ESXi host used.
a Storage vMotion never caused any issues when the disk was residing on our production storage array.
the corruption only happened when the virtual machine was stored on particular datastores on our DR storage array.

Now it got really 'interesting'. The thing is that our DR storage array has two separate storage controllers running in active-active mode. However, the LUNs are always owned by a particular controller. Although the other controller can take over from the controller who 'owns' the LUNs in case of a failure, the owner will process the I/O when everything is fine. Particular LUNs are thus handled by a particular controller.

So first I made a table where I listed the controllers and the LUNs it had ownership over, like this:

            Owner       
Controller      a               b
            LUN001          LUN002
            LUN003          LUN004
            LUN005          LUN006

Then I started to perform Storage vMotions of the ZFS disk from one LUN to the other. After performing several test, the pattern became quite obvious.

            LUN001  ->  LUN002  =   BAD
            LUN001  ->  LUN004  =   BAD
            LUN004  ->  LUN003  =   BAD
            LUN003  ->  LUN005  =   GOOD
            LUN005  ->  LUN001  =   GOOD

I continued to test some additional permutations but it became clear that only LUNs owned by controller b caused problems.

With the evidence in hand, I managed to convince our vendor support to replace storage controller b and that indeed resolved the problem. Data corruption due to a Storage vMotion never occurred after the controller was replaced.

There is no need to name/shame the vendor in this regard. The thing is that all equipment can fail and what can happen will happen. What really counts is: are you prepared?

RAID 5 is perfectly fine for home usage

2016-09-08T12:00:00+02:00

RAID 5 gets a lot of flak these days. You either run RAID 1, RAID 10 or you use RAID 6, but if you run RAID 5 you're told that you are a crazy person.

Using RAID 5 is portrayed as an unreasonable risk to the availability of your data. It is suggested that it is likely that you will lose your RAID array at some point.

That's an unfair representation of the actual risk that surrounds RAID 5. As I see it, the scare about RAID 5 is totally blown out of proportion.

I would argue that for small RAID arrays with a maximum of five to six drives, it's totally reasonable to use RAID 5 for your home NAS.

As far as I can tell, the campaign against RAID 5 mainly started with this article from zdnet.

As you know RAID 5 can tollerate a single drive failure. If a second drive dies and the first drive was not yet replaced or rebuild, you lose all contents of the array.

In the article the author argues that because drives become bigger but not more reliable, the risk of losing a second drive during a rebuild is so high that running RAID 5 is becoming risky.

You don't need a second drive failure for you to lose your data. A bad sector, also known as an Unrecoverable Read Error (URE), can also cause problems during a rebuild. Depending on the RAID implementation, you may lose some files or the entire array.

The author calculates and argues that the risk of such a bad sector or URE is so high with modern high-capacity drives, that this risk of a second drive failure during rebuild is almost unavoidable.

Most drives have a URE specification of 1 bit error in 12.5 TB of data (10^14). That number is used as an absolute, it's what drives do experience in our daily lives, but that's not true.

It's a worst-case number. You will see a read error in at-most 10^14 bits, but in practice drives are way more reliable.

I run ZFS on my 71 TB ZFS NAS and I scrub from time to time.

If that worst-case number were 'real', I would have caught some data errors by now. However, in line with my personal experience, ZFS hasn't corrected a single byte since the system came online a few years ago.

And I've performed so many scrubs that my system has read over a petabyte of data. No silent data corruption, no regular bad sectors.

It seems to me that all those risk aren't nearly as high as it seems.

I would argue that choosing RAID-5/Z in the right circumstances is reasonable. RAID-6 is clearly safer than RAID-5 as you can survive the loss of two drives instead of a single drive, but that doesn't mean that RAID-5 is unsafe.

If you are going to run a RAID 5 array, make sure you run a scrub or patrol read or whatever the name is that your RAID solution uses. A scrub is nothing more than attempt to try and read all data from disk.

Scrubbing allows detection of bad sectors in advance, so you can replace drives before they cause real problems (like failing during a rebuild).

If you keep the number of drives in a RAID-5 array low, maybe at most 5 or 6, I think for home users, who need to find a balance between cost and capacity, RAID-5 is an acceptable option.

And remember: if you care about your data, you need a backup anyway.

This topic was also discussed on reddit.

ZFS: resilver performance of various RAID schemas

2016-01-31T12:00:00+01:00

When building your own DIY home NAS, it is important that you simulate and test drive failures before you put your important data on it. It makes sense to know what to do in case a drive needs to be replaced. I also recommend putting a substantial amount of data on your NAS and see how long a resilver takes just so you know what to expect.

There are many reports of people building their own (ZFS-based) NAS who found out after a drive failure that resilvering would take days. If your chosen redundancy level for the VDEV would not protect against a second drive failure in the same VDEV (Mirror, RAID-Z) things may get scary. Especially because drives are quite bussy rebuilding data and the extra load on the remaining drives may increase the risk of a second failure.

The chosen RAID level for your VDEV, has an impact on the resilver performance. You may chose to accept lower resilver performance in exchange for additional redundancy (RAID-Z2, RAID-Z3).

I did wonder though how much those resilver times would differ between the various RAID levels. This is why I decided to run some tests to get some numbers.

Test hardware

I've used some test equipment running Debian Jessie + ZFS on Linux. The hardware is rather old and the CPU may have an impact on the results.

CPU : Intel(R) Core(TM)2 Duo CPU     E7400  @ 2.80GHz
RAM : 8 GB
HBA : HighPoint RocketRaid 2340 (each drive in a jbod)
Disk: Samsung Spinpoint F1 - 1 TB - 7200 RPM ( 12 x )

Test method

I've created a script that runs all tests automatically. This is how the script works:

Create pool + vdev(s).
Write data on pool ( XX % of pool capacity)
Replace arbitrary drive with another one.
Wait for resilver to complete.
Log resilver duration o csv file.

For each test, I fill the pool up to 25% with data before I measure resilver performance.

Caveats

The problem with the pool only being filled for 25% is that drives are fast at the start, but their performance deteriorates significantly as they fill up. This means that you cannot extrapolate the results and calculate resilver times for 50% or 75% pool usage, the numbers are likely worse than that.

I should run the test again with 50% usage to see if we can demonstrate this effect.

Beware that this test method is probably only suitable for DIY home NAS builds. Production file systems used within businesses may be way more fragmented and I've been told that this could slow down resilver times dramatically.

Test result (lower is better)

The results can only be used to demonstrate the relative resilver performance differences of the various RAID levels and disk counts per VDEV.

You should not expect the same performance results for your own NAS as the hardware probably differs significantly from my test setup.

Observations

I think the following observations can be made:

Mirrors resilver the fastest even if the number of drives involved is increased.
RAID-Z resilver performance is on-par with using mirrors when using 5 disks or less.
RAID-Zx resilver performance deteriorates as the number of drives in a VDEV increases.

I find it interesting that with smaller number of drives in a RAID-Z VDEV, rebuild performance is roughly on par with a mirror setup. If long rebuild times would scare you away from using RAID-Z, maybe it should not. There may be other reasons why you might shy away from RAID-Z, but this doesn't seem one of them.

RAID-Z2 is often very popular amongst home NAS builders, as it offers a very nice balance between capacity and redundancy. Wider RAID-Z2 VDEVs are more space efficient, but it is also clear that resilver operations take longer. Because RAID-Z2 can tollerate the loss of two drives, I think longer resilver times seem like a reasonable tradeoff.

It is clear that as you put more disks in a single RAID-Zx VDEV, rebuild times increase. This can be used as an argument to keep the number of drives per VDEV 'reasonable' or to switch to RAID-Z3.

25% vs 50% pool usage

To me, there's nothing special to see here. The resilver times are on average slightly worse than double the 25% resilver durations. As disks performance start to deteriorate as they fill up (inner tracks are shorter/slower) sequential performance drops. So this is why I would explain the results are slightly worse than perfect linear scaling.

Final words

I hope this benchmark is of interest to anyone and more importantly, you can run your own by using the aforementioned script. If you ever want to run your own benchmarks, expect the script to run for days. Leave a comment if you have questions or remarks about these test results or the way testing is done.

The 'hidden' cost of using ZFS for your home NAS

2016-01-02T12:00:00+01:00

Introduction

Update December 2023: In June, it was announced that iXsystems would sponsor implementing the VDEV expansion feature. A new pr has been created for this effort. The feature was merged into the code base, but may not be available to the general public before the end of 2024.

Many home NAS builders consider using ZFS for their file system. But there is a caveat with ZFS that people should be aware of.

Although ZFS is free software, implementing ZFS is not free. The key issue is that expanding capacity with ZFS is more expensive compared to legacy RAID solutions.

With ZFS, you either have to buy all storage you expect to need upfront, or you will be wasting a few hard drives on redundancy you don't need.

This fact is often overlooked, but it's very important to take it in consideration when planning a NAS build.

Other software RAID solutions like Linux MDADM lets you grow an existing RAID array with one disk at a time. This is also true for many hardware-based RAID solutions¹. This is ideal for home users because you can expand on a per-need basis.

ZFS does not allow this!

To understand why using ZFS may cost you extra money, we will dig a little bit into ZFS itself.

Quick recap of ZFS

The schema below illustrates the architecture of ZFS. There are a few things you should take away from it.

The main takeaway of this picture is that your ZFS pool and thus your file system is based on one or more VDEVs. And those VDEVs contain the actual hard drives.

Fault-tolerance or redundancy is addressed within a VDEV. A VDEV is either a mirror (RAID-1), RAIDZ (RAID-5) or RAIDZ2 (RAID-6)².

So it's important to understand that a ZFS pool itself is not fault-tolerant. If you lose a single VDEV within a pool, you lose the whole pool. You lose the pool, all data is lost.

You can't add hard drives to a VDEV

Now it's very important to understand that you cannot add hard drives to a VDEV.

This is the key limitation of ZFS as seen from the perspective of home NAS builders.

To expand the storage capacity of your pool, you need to add extra VDEVs. And because each VDEV needs to take care of its own redundancy, you also need to buy extra drives for parity.

I will quickly add that there is a way out: replace every hard drive in the VDEV, one by one, with a higher capacity hard drive. You will have to 'rebuild' or 'resilver' the VDEV after each replacement, but it will work, although it's a bit cumbersome and quite expensive.

So back to the topic at hand: what does this limitation mean in real life? I'll give an example.

Let's say you plan on building a small NAS with a capacity of four drives. Please don't create a three-drive RAID-Z thinking you can just add the fourth drive when you need to, because that's not possible.

In this example, you would be better off buying the fourth drive upfront and create a four-drive RAID-Z. This is an example where you are forced to buy the extra space you don't need yet upfront because expanding is otherwise not possible.

You could have expanded your pool with another VDEV consisting of a minimum of three drives (if you run RAID-Z) but the chassis has only room for one extra drive so that doesn't work.

Planning your ZFS Build with the VDEV limitation in mind

Many home NAS builders use RAID-6 (RAID-Z2) for their builds, because of the extra redundancy. This makes sense because a double drive failure is not something unheard of, especially during rebuilds where all drives are being taxed quite heavily for many hours.

I personally would recommend running RAID-Z2 over RAID-Z1 if you go over five to six drives and to spend the extra money on the additional hard drive it requires. Actually, With RAID-Z2 or RAID-6, I think it's perfectly reasonable to run a single VDEV at home with up to 12 drives³.

With RAID-Z2 however, the 'ZFS tax' is even more clearly visible. By having to add an additional VDEV, you will also lose two drives due to parity overhead.

Please note that the 'yellow' drives mark the parity/redundancy overhead. It does not mark where parity data lives (it's striped across all drives).

Let's illustrate the above picture with an example. Your NAS chassis can hold a maximum of twelve drives. You start out with six drives in a RAID-Z2. At some point you want to expand. The cheapest option is to expand with another RAID-Z2 consisting of four drives (minimum size of a RAID-Z2 VDEV).

With a cost of $150 per hard drive⁵, expanding the capacity of your pool will cost you $600 instead of $150 (single drive) and $300 dollar of the $600 (50%) is wasted on redundancy you don't really need.

Furthermore, you can no longer expand your pool, so the remaining two drive slots are 'wasted'⁴. You end up with a maximum of ten drives.

In this example, to make use of the drive capacity of your NAS chassis, you should expand with another six hard drives. That would cost you $900 and $300 of that $900 (33%) is wasted on redundancy. This is illustrated above.

Storage-wise it's more efficient to expand with six drives instead of four. But it will cost you another $300 to expand, paying for storage you may not immediately need.

But both options aren't that efficient. Because you end up using four drives for parity where two would - in my view - be sufficient.

So, if you want to get the most capacity out of that chassis, and the most space per dollar, your only option is to buy all twelve drives upfront and create a single RAID-Z2 consisting of twelve drives.

Buying all drives upfront is expensive and you may only benefit from that extra space years down the road.

Summary

So I hope this example clearly illustrates the issue at hand. With ZFS, you either need to buy all storage upfront or you will lose hard drives to redundancy you don't need, reducing the maximum storage capacity of your NAS.

You have to decide what your needs are. ZFS is an awesome file system that offers you way better data integrity protection than other file system + RAID solution combination.

But implementing ZFS has a certain 'cost'. You must decide if ZFS is worth it for you.

Update April 2023: It has been fairly quiet since the announcement of RAIDZ expansion. The Github PR about this feature is rather stale and people are wondering what the status is and what the plans are. Meanwhile, FreeBSD has announced In February 2023 that they suspect to integrate RAIDZ expansion by Q3.

Update June 2021| It seems that RAIDZ expansion is now being worked on. It will probably be available somewhere around August 2022.

I have written a blogpost about this new feature. The bad news is that adding drives to an existing vdev may accrue some overhead, but the good news is that this overhead can be recovered.

Update October 2017 | Please note that RAIDZ expansion is under development.

RAIDZ expansion (most requested ZFS feature ever?) is coming, courtesy of @freebsdfndation. Sneak preview at OpenZFS DevSummit!
— OpenZFS (@OpenZFS) October 19, 2017

Addressing some feedback

I found out that my article was discussed on a vodcast of BSDNOW.

This article also got some attention on hacker news.

To me, some of the feedback is not 'wrong' but feels rather disingenuous or not relevant for the intended audience of this article. I have provided the links so you can make up your own mind.

This article has a particular user group in mind so you really should think about how much their needs align with yours.

You are steering people away from ZFS

No I don't and this is not my intention. I run ZFS myself on two servers. I do feel that sometimes the downsides of ZFS are wiped under the rug and we should be very open and clear about them towards people seeking advice.

Use mirrors not RAID-Z(2/3)!

Doesn't make much sense to me for home NAS builders.

Using mirrors is wasting space

Advising people to use mirrors instead of RAID-Z(2/3) I do find a little bit disingenuous. Because you are throwing away 50% of your disk capacity. With RAIDZ you 'lose' 33% for three drives, 25% for four drives. If we look at RAIDZ2, we would 'lose' 33% for six drives, 25% for eight drives and only 20% for ten drives.

In the end, you are waisting multiple drives worth of storage capacity depending on the number of drives in your pool.

Adding mirrors with larger drives

As time goes by, larger disks become cheaper. So it could make sense to expand your pool with mirrors based on bigger drives than the original drives you started out on. The size of your pool would increase. However, it's still only 50% space efficient.

Random I/O performance is better

Using mirrors is running RAID 10. Yes you can expand your pool with two drives at a time, and you gain better random I/O performance. However, the large majority of home NAS builders don't care about random I/O performance. You just care if you can saturate gigabit and have one big pool of storage. In that case, you don't need the random IOPs.

If you run some VMs from your storage that require high storage performance, it's an entirely different matter. But I expect that most DIY NAS builders just want some storage to put a ton of data on and nothing more.

RAIDZ2 is more reliable than using mirrors

The redundancy of RAIDZ2 beats using mirrors. (If during a rebuild the surviving member of a mirror fails (the one disk in the pool that is taxed the most during rebuild) you lose your pool. With RAIDZ2 any second drive can fail and you are still OK.

There is only one 'upside' regarding mirrors that is discussed in the next section.

Mirror rebuild times are better

The only upside of using mirrors is that in the event a disk has failed and the new disk is being 'resilvered' it is reported that those rebuilds tend to be faster than if you use RAID-Z(2/3). I think this is no different from legacy RAID, where the main difference with ZFS is that ZFS only rebuilds actual data, not the entire disk.

ZFS rebuilds are faster

This is indeed a benefit of ZFS. The question is how relevant it is for you.

ZFS only rebuilds data. Legacy RAID just rebuilds every 'bit' on a drive. The latter takes longer than the former. So with legacy RAID, rebuild times depend on the size of a single drive, not on the number of drives in the array, no matter how much data you have stored on your array.

My old 18 TB server was based on a single twenty-drive RAID 6 using MDADM. It took 5 hours to rebuild a 1 TB drive. If you would have used 4 TB drives, it would have taken 20 hours if I'm allowed to extrapolate. With ZFS - if you would have been using only 50% of capacity - those rebuild times would have been half of this.

Personally with RAID6 or with RAIDZ2, rebuild times aren't that a big of a deal as you can lose a second drive and still be safe.

Just replace existing drives with bigger ones!

I did briefly touch this option in the article above. I will address it again. The problem with this approach is twofold. First, you can't expand storage capacity as you need it. You need to replace all existing drives with larger ones.

The procedure itself is also a bit cumbersome and time intensive. You need to replace each drive one by one. And every time, you need to 'resilver' your VDEV. Only when all drives have been replaced you will be able to grow the size of your pool.

If you are OK with this approach - and people have used it - it is a way to work around the 'ZFS-tax'.

Not using ZFS is putting your data at great risk!

The BSDNOW podcasts seems to agree with me that if you want true data safety, this 'ZFS-tax' is just the price you have to pay. Either you go with mirrors or you accept the extra parity redundancy.

It is not my goal to steer you away from ZFS. The above is true. ZFS offers something no other (stable) file system currently offers to home NAS builders. But at a cost.

The thing is that I find it perfectly reasonable for home NAS users to just buy a Synology, QNAP or some ready-made NAS from another quality brand. That's what the majority of people do and I think it's a reasonable option. I don't think you are taking crazy risks if you would do so.

If you do build your own home NAS, it's reasonable to accept the 'risk' of using Windows with storage spaces or hardware RAID. Or using Linux with MDADM or hardware RAID. I would say: ZFS is clearly technically the better option, but those 'legacy' options are not so bad that you are taking unreasonable risks with your data.

So using ZFS is the better option, it's up to you and your particular needs and circumstances to decide if using ZFS is worth it for you.

I believe hardware-based RAID is 100% dead, especially with SSDs but historically speaking hardare RAID allowed for flexible expansion. ↩
It can even use tripple parity (RAID-Z3) but I doubt many of you will ever need that. ↩
For my own 71 TB storage NAS I decided at that time to run with an eighteen-disk VDEV plus a six-disk VDEV. Not standard, but I decided that I accept the risk. ↩
Expanding with a VDEV consisting of a mirrored pair is technically possible but it breaks the RAID-Z2 redundancy. It doesn't make much sense to me. ↩
Just an example for illustration purposes. ↩

ZFS performance on HP Proliant Microserver Gen8 G1610T

2015-08-14T12:00:00+02:00

I think the HP Proliant Microserver Gen8 is a very interesting little box if you want to build your own ZFS-based NAS. The benchmarks I've performed seem to confirm this.

The Microserver Gen8 has nice features such as:

iLO (KVM over IP with dedicated network interface)
support for ECC memory
2 x Gigabit network ports
Free PCIe slot (half-height)
Small footprint
Fairly silent
good build quality

The Microserver Gen8 can be a better solution than the offerings of - for example - Synology or QNAP because you can create a more reliable system based on ECC-memory and ZFS.

Please note that the G1610T version of the Microserver Gen8 does not ship with a DVD/CD drive as depicted in the image above.

The Gen8 can be found fairly cheap on the European market at around 240 Euro including taxes and if you put in an extra 8 GB of memory on top of the 2 GB installed you have a total of 10 GB, which is more than enough to support ZFS.

The Gen8 has room for 4 x 3.5" hard drives so with todays large disk sizes you can pack quite a bit of storage inside this compact machine.

Netto storage capacity:

This table gives you a quick overview of the netto storage capacity you would get depending on the chosen drive size and redundancy.

Drive size	RAIDZ	RAIDZ2 or Mirror
3 TB	9 TB	6 TB
4 TB	12 TB	8 TB
6 TB	18 TB	12 TB
8 TB	24 TB	16 TB

Boot device

If you want to use all four drive slots for storage, you need to boot this machine from either the fifth internal SATA port, the internal USB 2.0 port or the microSD card slot.

The fifth SATA port is not bootable if you disable the on-board RAID controller and run in pure AHCI mode. This mode is probably the best mode for ZFS as there seems to be no RAID controller firmware active between the disks and ZFS. However, only the four 3.5" drive bays are bootable.

The fifth SATA port is bootable if you configure SATA to operate in Legacy mode. This is not recommended as you lose the benefits of AHCI such as hot-swap of disks and there are probably also performance penalties.

The fifth SATA port is also bootable if you enable the on-board RAID controller, but do not configure any RAID arrays with the drives you plan to use with ZFS (Thanks Mikko Rytilahti). You do need to put the boot drive in a RAID volume in order to be able to boot from the fifth SATA port.

The unconfigured drives will just be passed as AHCI devices to the OS and thus can be used in your ZFS array. The big question here is what happens if you encounter read errors or other drive problems that ZFS could handle, but would be a reason for the RAID controller to kick a drive off the SATA bus. I have no information on that.

I myself used an old 2.5" hard drive with a SATA-to-USB converter which I stuck in the case (use double-sided tape or velcro to mount it to the PSU). Booting from USB stick is also an option, although a regular 2.5" hard drive or SSD is probably more reliable (flash wear) and faster.

Boot performance

The Microserver Gen8 takes about 1 minute and 50 seconds just to pass the BIOS boot process and start booting the operating system (you will hear a beep).

Test method and equipment

I'm running Debian Jessie with the latest stable ZFS-on-Linux 0.6.4. Please note that reportedly FreeNAS also runs perfectly fine on this box.

I had to run my tests with the disk I had available:

root@debian:~# show disk -sm
-----------------------------------
| Dev | Model              | GB   |   
-----------------------------------
| sda | SAMSUNG HD103UJ    | 1000 |   
| sdb | ST2000DM001-1CH164 | 2000 |   
| sdc | ST2000DM001-1ER164 | 2000 |   
| sdd | SAMSUNG HM250HI    | 250  |   
| sde | ST2000DM001-1ER164 | 2000 |   
-----------------------------------

The 250 GB is a portable disk connected to the internal USB port. It is used as the OS boot device. The other disks, 1 x 1 TB and 3 x 2 TB are put together in a single RAIDZ pool, which results in 3 TB of storage.

Tests with 4-disk RAIDZ VDEV

root@debian:~# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool  48.8G  2.54T  48.8G  /testpool
root@debian:~# zpool status
  pool: testpool
 state: ONLINE
  scan: none requested
config:

    NAME                        STATE     READ WRITE CKSUM
    testpool                    ONLINE       0     0     0
      raidz1-0                  ONLINE       0     0     0
        wwn-0x50000f0008064806  ONLINE       0     0     0
        wwn-0x5000c5006518af8f  ONLINE       0     0     0
        wwn-0x5000c5007cebaf42  ONLINE       0     0     0
        wwn-0x5000c5007ceba5a5  ONLINE       0     0     0

errors: No known data errors

Because a NAS will face data transfers that are sequential in nature, I've done some tests with 'dd' to measure this performance.

Read performance:

root@debian:~# dd if=/testpool/test.bin of=/dev/null bs=1M
50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 162.429 s, 323 MB/s

Write performance:

root@debian:~# dd if=/dev/zero of=/testpool/test.bin bs=1M count=50000 conv=sync 50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 169.572 s, 309 MB/s

Test with 3-disk RAIDZ VDEV

After the previous test I wondered what would happen if I would exclude the older 1 TB disk and create a pool with just the 3 x 2 TB drives. This is the result:

Read performance:

root@debian:~# dd if=/testpool/test.bin of=/dev/null bs=1M conv=sync 50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 149.509 s, 351 MB/s

Write performance:

root@debian:~# dd if=/dev/zero of=/testpool/test.bin bs=1M count=50000 conv=sync 50000+0 records in 50000+0 records out 52428800000 bytes (52 GB) copied, 144.832 s, 362 MB/s

The performance is clearly better even there's one disk less in the VDEV. I would have liked to test with an additional 2 TB drive what kind of performance would be achieved with four drives but I only have three.

The result does show that the pool is more than capable of sustaining gigabit network transfer speeds.

This is confirmed when performing the actual network file transfers. In the example below, I simulate a copy of a 50 GB test file from the Gen8 towards a test system using NFS. Tests are performed using the 3-disk pool.

NFS read performance:

root@nano:~# dd if=/mnt/server/test2.bin of=/dev/null bs=1M
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 443.085 s, 118 MB/s

NFS write performance:

root@nano:~# dd if=/dev/zero of=/mnt/server/test2.bin bs=1M count=50000 conv=sync 
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 453.233 s, 116 MB/s

I think these results are excellent. Tests with the 'cp' command give the same results.

I've also done some test with the SMB/CIFS protocol. I've used a second Linux box as a CIFS client to connect to the Gen8.

CIFS read performance:

root@nano:~# dd if=/mnt/test/test.bin of=/dev/null bs=1M
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 527.778 s, 99.3 MB/s

CIFS write performance:

root@nano:~# dd if=/dev/zero of=/mnt/test/test3.bin bs=1M count=50000 conv=sync
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 448.677 s, 117 MB/s

Hot-swap support

Although it's even printed on the hard drive caddies that hot-swap is not supported, it does seem to work perfectly fine if you run the SATA controller in AHCI mode.

Fifth SATA port for SSD SLOG/L2ARC?

If you buy a converter cable that converts a floppy power connector to a SATA power connector, you could install an SSD. This SSD can then be used as a dedicated SLOG device and/or L2ARC cache if you have a need for this.

RAIDZ, is that OK?

If you want maximum storage capacity with redundancy RAIDZ is the only option. RAID6 or two mirrored VDEVs is more reliable, but will reduce available storage space by a third.

The main risk of RAIDZ is a double-drive failure. As with larger drive sizes, a resilver of a VDEV will take quite some time. It could take more than a day before the pool is resilvered, during which you run without redundancy.

With the low number of drives in the VDEV the risk of a second drive failure may be low enough to be acceptable. That's up to you.

Noise levels

In the past, there have been reports about the Gen8 making tons of noise because the rear chasis fan spins at a high RPM if the RAID card is set to AHCI mode.

I myself have not encountered this problem. The machine is almost silent.

Power consumption

With drives spinning: 50-55 Watt. With drives standby: 30-35 Watt.

Conclusion

I think my benchmarks show that the Microserver Gen8 could be an interesting platform if you want to create your own ZFS-based NAS.

Please note that it is likely that since the Gen9 server platform is already out for some time, HP may release a Gen9 version of the microserver in the near future. However as of August 2015, there is no information on this yet and it is not clear if a successor is going to be released.

The sorry state of CoW file systems

2015-03-01T12:00:00+01:00

I'd like to argue that both ZFS and BTRFS both are incomplete file systems with their own drawbacks and that it may still be a long way off before we have something truly great.

Both ZFS and BTRFS are two heroic feats of engineering, created by people who are probably ten times more capable and smarter than me. There is no question about my appreciation for these file systems and what they accomplish.

Still, as an end-user, I would like to see some features that are often either missing or not complete. Make no mistake, I believe that both ZFS and BTRFS are probably the best file systems we have today. But they can be much better.

I want to start with a terse and quick overview on why both ZFS and BTRFS are such great file systems and why you should take some interest in them.

Then I'd like to discuss their individual drawbacks and explain my argument.

Why ZFS and BTRFS are so great

Both ZFS and BTRFS are great for two reasons:

They focus on preserving data integrity
They simplify storage management

Data integrity

ZFS and BTRFS implement two important techniques that help preserve data.

Data is checksummed and its checksum is verified to guard against bit rot due to broken hard drives or flaky storage controllers. If redundancy is available (RAID), errors can even be corrected.
Copy-on-Write (CoW), existing data is never overwritten, so any calamity like sudden power loss cannot cause existing data to be in an inconsistent state.

Simplified storage management

In the old days, we had MDADM or hardware RAID for redundancy. LVM for logical volume management and then on top of that, we have the file system of choice (EXT3/4, XFS, REISERFS, etc).

The main problem with this approach is that the layers are not aware of each other and this makes things very inefficient and more difficult to administer. Each layer needs it's own attention.

For example, if you simply want to expand storage capacity, you need to add drives to your RAID array and expand it. Then, you have to alert the LVM layer of the extra storage and as a last step, grow the file system.

Both ZFS and BTRFS make capacity expansion a simple one line command that addresses all three steps above.

Why are ZFS and BTRFS capable of doing this? Because they incorporate RAID, LVM and the file system in one single integrated solution. Each 'layer' is aware of the other, they are tightly integrated. Because of this integration, rebuilds after a drive faillure are often faster than with 'legacy RAID' solutions, because they only need to rebuild the actual data, not the entire drive.

And I'm not even talking about the joy of snapshots here.

The inflexibility of ZFS

The storage building block of ZFS is a VDEV. A VDEV is either a single disk (not so interesting) or some RAID scheme, such as mirroring, single-parity (RAIDZ), dual-parity (RAIDZ2) and even tripple-parity (RAIDZ3).

To me, a big downside to ZFS is the fact that you cannot expand a VDEV. Ok, the only way you can expand the VDEV is quite convoluted. You have to replace all of the existing drives, one by one, with bigger ones and rebuild the VDEV each time you replace one of the drives. Then, when all drives are of the higher capacity, you can expand your VDEV. This is quite impractical and time-consuming, if you ask me.

ZFS expects you just to add extra VDEVS. So if you start with a single 6-drive RAIDZ2 (RAID6), you are expected to add another 6-drive RAIDZ2 if you want to expand capacity.

What I would want to do is just to ad one or two more drives and grow the VDEV, as is possible with many hardware RAID solutions and with "MDADM --grow" for ages.

Why do I prefer this over adding VDEVS? Because it's quite evident that this is way more economical. If I can just expand my RAIDZ2 from 6 drives to 12 drives, I would only sacrifice two drives for parity. If I add two VDEVS each of them RAIDZ2, I sacrifice four drives (16% vs 33% capacity loss).

I can imagine that in the enterprise world, this is just not that big of a deal, a bunch of drives are a rounding error on the total budget and availability and performance are more important. Still, I'd like to have this option.

Either you are forced to buy and implement the storage you may expect to need in the future, or you must add it later on, wasting drives on parity you would otherwise not have done.

Maybe my wish for a zpool grow option is more geared to hobbyist or home usage of ZFS and ZFS was always focussed on enterprise needs, not the needs of hobbyists. So I'm aware of the context here.

I'm not done with ZFS however, because the way ZFS works, there is another great inflexibility. If you don't put the 'right' number of drives in a VDEV, you may lose significant portions of storage, which is a side-effect of how ZFS works.

The following ZFS pool configurations are optimal for modern 4K sector harddrives:
RAID-Z: 3, 5, 9, 17, 33 drives
RAID-Z2: 4, 6, 10, 18, 34 drives
RAID-Z3: 5, 7, 11, 19, 35 drives

I've seen first-hand with my 71 TiB NAS that if you don't use the optimal number of drives in a VDEV, you may lose whole drives worth of netto storage capacity. In that regard, my 24-drive chassis is very suboptimal.

The sad state of RAID on BTRFS

BTRFS has none of the downsides of ZFS as described in the previous section as far as I'm aware of. It has plenty of its own, though. First of all: BTRFS is still not stable, especially the RAID 5/6 part is unstable.

The RAID 5 and RAID 6 implementation are so new, the ink they were written with is still wet (February 8th 2015). Not something you want to trust your important data to I suppose.

I did setup a test environment to play a bit with this new Linux kernel (3.19.0) and BTRFS to see how it works and although it is not production-ready yet, I really like what I see.

With BTRFS you can just add or remove drives to a RAID6 array as you see fit. Add two? Subtract 3? Whatever, the only thing you have to wait for is BTRFS rebalancing the data over either the new or remaining drives.

This is friggin' awesome.

If you want to remove a drive, just wait for BTRFS to copy the data from that drive to the other remaining drives and you can remove it. You want to expand storage? Just add the drives to your storage pool and have BTRFS rebalance the data (which may take a while, but it works).

But I'm still a bit sad. Because BTRFS does not support anything beyond RAID6. No multiple RAID6 (RAID60) arrays or tripple-parity, as ZFS supports for ages. As with my 24-drive file server, putting 24 drives in a single RAID6, starts to feel like I'm asking for trouble. Tripple-parity or RAID 60 would probably be more reasonable. But no luck with BTRFS.

However, what really frustrates me is this article by Ronny Egner. The author of snapraid, Andrea Mazzoleni, has written a functional patch for BTRFS that implements not only tripple-parity RAID, but even up to six parity disks for a volume.

The maddening thing is that the BTRFS maintainers are not planning to include this patch into the BTRFS code base. Please read Ronny's blog. The people working on BTRFS are working for enterprises who want enterprise features. They don't care about tripple-parity or features like that because they have access to something presumably better: distributed file systems, which may do away with the need for larger disk arrays and thus tripple-parity.

BTRFS is in development for a very long time and only recently has RAID 5/6 support been introduced. The risk of the write-hole, something addressed by ZFS ages ago, is still an open issue. Considering all of this, BTRFS is still a very long way off, of being the file system of choice for larger storage arrays.

BTRFS seems to be way more flexible in terms of storage expansion or shrinking, but it slow pace of development makes it still unusable for anything serious for at least the next year I guess.

Conclusion

BTRFS addresses all the inflexibilities of ZFS but it's immaturity and lack of more advanced RAID schemes makes it unusable for larger storage solutions. This is so sad because by design it seems to be the better, way more flexible option as compared to ZFS.

I do understand the view of the BTRFS developers. With the enterprise data sets, at scale, it's better to use distributed file systems to handle storage and redundancy, than on the smaller system scale. But this kind of environment is not reachable for many.

So at the moment, compared to BTRFS, ZFS is still the better option for people who want to setup large, reliable storage arrays.

Configuring SCST iSCSI target on Debian Linux (Wheezy)

2015-02-01T12:00:00+01:00

My goal is to export ZFS zvol volumes through iSCSI to other machines. The platform I'm using is Debian Wheezy.

There are three iSCSI target solutions available for Linux:

I've briefly played with LIO but the targetcli tool is interactive only. If you want to automate and use scripts, you need to learn the Python API. I wonder what's wrong with a plain old text-based configuration file.

iscsitarget or IET is broken on Debian Wheezy. If you just 'apt-get install iscsitarget', the iSCSI service will just crash as soon as you connect to it. This has been the case for years. I wonder why they don't just drop this package. It is true that you can manually download the "latest" version of IET, but don't bother, it seems abandoned. The latest release stems from 2010.

It seems that SCST is at least maintained and uses plain old text-based configuration files. So it has that going for it, which is nice. SCST does not require kernel patches to run. But particularly a patch regarding "CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION" is said to improve performance.

To use full power of TCP zero-copy transmit functions, especially
dealing with user space supplied via scst_user module memory, iSCSI-SCST
needs to be notified when Linux networking finished data transmission.
For that you should enable CONFIG_TCP_ZERO_COPY_TRANSFER_COMPLETION_NOTIFICATION
kernel config option. This is highly recommended, but not required.
Basically, iSCSI-SCST works fine with an unpatched Linux kernel with the
same or better speed as other open source iSCSI targets, including IET,
but if you want even better performance you have to patch and rebuild
the kernel.

So in general, patching your kernel is not always required, but an example will be given anyway.

Getting the source

cd /usr/src

We need the following files:

wget http://heanet.dl.sourceforge.net/project/scst/scst/scst-3.0.0.tar.bz2
wget http://heanet.dl.sourceforge.net/project/scst/iscsi-scst/iscsi-scst-3.0.0.tar.bz2
wget http://heanet.dl.sourceforge.net/project/scst/scstadmin/scstadmin-3.0.0.tar.bz2

We extract them with:

tar xjf scst-3.0.0.tar.bz2
tar xjf iscsi-scst-3.0.0.tar.bz2
tar xjf scstadmin-3.0.0.tar.bz2

Patching the kernel

You can skip this part if you don't feel like you need or want to patch your kernel.

apt-get install linux-source kernel-package

We need to extract the kernel source:

cd /usr/src
tar xjf linux-source-3.2.tar.bz2
cd linux-source-3.2

Now we first copy the kernel configuration from the current system:

cp /boot/config-3.2.0-4-amd64 .config

We patch the kernel with two patches:

patch -p1 < /usr/src/scst-3.0.0/kernel/scst_exec_req_fifo-3.2.patch
patch -p1 < /usr/src/iscsi-scst-3.0.0/kernel/patches/put_page_callback-3.2.57.patch

It seems that for many different kernel versions, separate patches can be found in the above paths. If you follow these steps at a later date, please check the version numbers.

The patches are based on stock kernels from kernel.org. I've applied the patches against the Debian-patched kernel and faced no problems, but your milage may vary.

Let's build the kernel (will take a while):

yes | make-kpkg -j $(nproc) --initrd --revision=1.0.custom.scst kernel_image

The 'yes' is piped into the make-kpkg command to answer some questions with 'yes' during compilation. You could also add the appropriate value in the .config file.

The end-result of this command is a kernel package in .deb format in /usr/src. Install it like this:

dpkg -i /usr/src/<custom kernel image>.deb

Now reboot into the new kernel:

reboot

Compiling SCST, ISCS-SCST and SCSTADMIN

cd /usr/src/scst-3.0.0
make install

cd /usr/src/iscsi-scst-3.0.0
make install

cd /usr/src/scstadmin-3.0.0
make install

Make SCST start at boot

On Debian Jessie:

systemctl enable scst.service

Configure SCST

Copy the example configuration file to /etc:

cp /usr/src/iscsi-scst-3.0.0/etc/scst.conf /etc

Edit /etc/scst.conf to your liking. This is an example:

HANDLER vdisk_fileio {
        DEVICE disk01 {
                filename /dev/sdb
                nv_cache 1
        }
}

TARGET_DRIVER iscsi {
        enabled 1

        TARGET iqn.2015-10.net.vlnb:tgt {
                IncomingUser "someuser somepasswordof12+chars"
                HeaderDigest   "CRC32C,None"
                DataDigest   "CRC32C,None"
                LUN 0 disk01

                enabled 1
        }
}

Please note that the password must be at least 12 characters.

After this, you can start the SCST module and connect your initiator to the appropriate LUN.

/etc/init.d/scst start

Closing words

It turned out that setting up SCST and compiling a kernel wasn't that much of a hassle. The main issue with patching kernels is that you have to repeat the procedure every time a new kernel version is released. And there is always a risk that a new kernel version breaks the SCST patches.

However, the whole process can be easily automated and thus run as a test in a virtual environment.

Why I Do Use ZFS as a File System for My NAS

2015-01-29T12:00:00+01:00

On February 2011, I posted an article about my motivations why I did not use ZFS as a file system for my 18 TB NAS.

You have to understand that at the time, I believe the arguments in the article were relevant, but much has changed since then, and I do believe this article is not relevant anymore.

My stance on ZFS is in the context of a home NAS build.

I really recommend giving ZFS a serious consideration if you are building your own NAS. It's probably the best file system you can use if you care about data integrity.

ZFS may only be available for non-Windows operating systems, but there are quite a few easy-to-use NAS distros available that turn your hardware into a full-featured home NAS box, that can be managed through your web browser. A few examples:

I also want to add this: I don't think it's wrong or particular risky if you - as a home NAS builder - would decide not to use ZFS and select a 'legacy' solution if that better suits your needs. I think that proponents of ZFS often overstate the risks ZFS mitigates a bit, maybe to promote ZFS. I do think those risks are relevant but it all depends on your circumstances. So you decide.

May 2016: I have also written a separate article on how I feel about using ZFS for DIY home NAS builds.

Arstechnica article about FreeNAS vs NAS4free.

If you are quite familiar with FreeBSD or Linux, I do recommend this ZFS how-to article from Arstechnica. It offers a very nice introduction to ZFS and explains terms like 'pool' and 'vdev'.

If you are planning on using ZFS for your own home NAS, I would recommend reading the following articles:

My historical reasons for not using ZFS at the time

When I started with my 18 TB NAS in 2009, there was no such thing as ZFS for Linux. ZFS was only available in a stable version for Open Solaris. We all know what happened to Open Solaris (it's gone).

So you might ask: "Why not use ZFS on FreeBSD then?". Good question, but it was bad timing:

The FreeBSD implementation of ZFS became only stable [sic] in January 2010, 6 months after I build my NAS (summer 2009). So FreeBSD was not an option at that time.

One of the other objections against ZFS is the fact that you cannot expand your storage by adding single drives and growing the array as your data set grows.

A ZFS pool consists of one or more VDEVs. A VDEV is a traditional RAID-array. You expand storage capacity by expanding the ZFS pool, not the VDEVS. You cannot expand the VDEV itself. You can only add VDEVS to a pool.

So ZFS either forces you to invest in storage you don't need upfront, or it forces you invest later on because you may waste quite a few extra drives on parity. For example, if you start with a 6-drive RAID6 (RAIDZ) configuration, you will probably expand with another 6 drives. So the pool has 4 parity drives on 12 total drives (33% loss). Investing upfront in 10 drives instead of 6 would have been more efficient because you only lose 2 drives out of 10 to parity (20% loss).

So at the time, I found it reasonable to stick with what I knew: Linux & MDADM.

But my new 71 TiB NAS is based on ZFS.

I wrote an article about my worry that ZFS may die with FreeBSD as it sole backing, but fortunately, I've been proven very, very wrong.

ZFS is now supported on FreeBSD and Linux. Despite some licencing issues that prevent ZFS from being integrated in the Linux kernel itself, it can still be used as a regular kernel module and it works perfectly.

There is even an open-source ZFS consortium that brings together all the developers for the different operating systems supporting ZFS.

ZFS is here to stay for a very long time.

FreeBSD 10.1 unattended install over PXE & HTTP (no NFS)

2015-01-16T12:00:00+01:00

To gain some more experience with FreeBSD, I decided to make a PXE-based unattended installation of FreeBSD 10.1.

My goal is to set something up similar to Debian/Ubuntu + preseeding or Redhat/CentOS + kickstart.

Getting a PXE-based unattended installation of FreeBSD 10.1 was not easy and I was unable to automate a ZFS-based install using bsdinstall.

I would expect someting like the netboot install

Below, I've documented what I've done to do a basic installation of FreeBSD using only DHCP, TFTP, no NFS required.

Overview of all the steps:

have a working DHCP with PXE boot options
have a working TFTP server
customise your pxelinux boot menu
install a FreeBSD box manually, or use an existing one
download and install mfsbsd on the FreeBSD system
download a FreeBSD release iso image on the FreeBSD system
configure and customise your FreeBSD PXE boot image settings
build the PXE boot image and copy it to your TFTP server
PXE boot your system and boot the FreeBSD image

Setting up a DHCP server + TFTP server

Please take a look at another article I wrote on setting up PXE booting.

Configuring the PXE boot menu

Add these lines to your PXE Menu:

LABEL FreeBSD10
kernel memdisk
append initrd=BSD/FreeBSD/10.1/mfsbsd-10.1-RC3-amd64.img harddisk raw

Setup or gain access to a FreeBSD host

You need to setup or gain access to a FreeBSD system, because the mfsbsd tool only works on FreeBSD. You will use this system to generate a FreeBSD PXE boot image.

Installing mfsbsd

First we download mfsbsd.

fetch http://mfsbsd.vx.sk/release/mfsbsd-2.1.tar.gz
tar xzf mfsbsd-2.1.tar.gz

Then we get a FreeBSD ISO:

fetch http://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/10.1/FreeBSD-10.1-RELEASE-amd64-disc1.iso

Mount the ISO:

mdconfig -a -t vnode -f /root/FreeBSD-10.1-RELEASE-amd64-disc1.iso
mount_cd9660 /dev/md0 /cdrom/

setup rc.local

Enter the mfsbsd-2.1 directory. Put the following content in the conf/rc.local file.

fetch http://<yourwebserver>/pxe/installerconfig -o /etc/installerconfig
tail -n 7 /etc/rc.local > /tmp/start.sh
chmod +x /tmp/start.sh
/tmp/start.sh 
exit 0

#!/bin/csh
setenv DISTRIBUTIONS "kernel.txz base.txz"
setenv BSDINSTALL_DISTDIR /tmp
setenv BSDINSTALL_DISTSITE
ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.1-RELEASE

bsdinstall distfetch 
bsdinstall script /etc/installerconfig

As you can see there is a script within a script that is executed separately by rc.local. That's a bit ugly but it does work.

setup installerconfig (FreeBSD unattended install)

The 'installerconfig' script is a script in a special format used by the bsdinstall tool to automate the installation. The top is used to control variables used during the unattended installation. The bottom is a script executed post-install chrooted on the new system.

Put this in 'installerconfig'

PARTITIONS=da0
DISTRIBUTIONS="kernel.txz base.txz"
BSDINSTALL_DISTDIR=/tmp
BSDINSTALL_DISTSITE=ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/10.1-RELEASE

#!/bin/sh
echo "Installation complete, running in host system"
echo "hostname=\"FreeBSD\"" >> /etc/rc.conf
echo "autoboot_delay=\"5\"" >> /boot/loader.conf
echo "sshd_enable=YES" >> /etc/rc.conf
echo "Setup done" >> /tmp/log.txt
echo "Setup done."
poweroff

As you can see, the post-install script enables SSH, sets the hostname and reduced the autoboot delay.

Please note that I faced an issue where the bsdinstall program would not interpret the options set in the installerconfig script. This is why I exported them with 'setenv' in the rc.local script.

With Debian preseeding or Redhat kickstarting, you can host the preseed or kickstart file on a webserver. Changing the PXE-based installation is just a matter of edditing the preseed or kickstart file on the webserver.

Because it's not fun having to generate a new image every time your want to update your unattended installation, it's recommended to host the installerconfig file on a webserver, as if it is a preseed or kickstart file.

This saves you from having to regenerate the PXE-boot image file every time.

You can still put the installer config in the image itself. If you want a fixed 'installerconfig' file containing the bsdinstall instructions, put this file also in the 'conf' directory. Next, edit the Makefile. Search for this string:

.for FILE in rc.conf ttys

For me, it was at line 315. Change it to:

.for FILE in rc.conf ttys installerconfig

Building the PXE boot image

Now everything is configured, we can generate the boot image with mfsbsd. Run 'make'. Then when it fails with this error:

Creating image file ...
/root/mfsbsd-2.1/tmp/mnt: write failed, filesystem is full
*** Error code 1

Stop.
make: stopped in /root/mfsbsd-2.1

just run 'make' again. In my experience, make worked the second time, consistently. I'm not sure why this happens.

The end result of this whole process is a file like 'mfsbsd-se-10.1-RC3-amd64.img'.

You can copy this image to the appropriate folder on your TFTP server. In my example it would be:

/srv/tftp/BSD/FreeBSD/10.1/mfsbsd-10.1-RC3-amd64.img

Test the PXE installation

Boot a test machine from PXE and boot your custom generated image.

Final words

I'm a bit unhappy about how difficult it was to create an PXE-based unattended FreeBSD installation. The bsdinstall installation software seems buggy to me. However, it could be just me: that I have misunderstood how it al works. However, I can't seem to find any documentation on how to properly use the bsdinstall system for an unattended installation.

If anyone has suggestions or ideas to implement an unattended bsdinstall script 'properly', with ZFS support, I'm all ears.

This is the recipe I tried to use to get a root-on-zfs install:

ZFSBOOT_POOL_NAME=TEST_ROOT
ZFSBOOT_VDEV_TYPE=mirror
ZFSBOOT_POOL_SIZE=10g
ZFSBOOT_DISKS="da0 da1"
ZFSBOOT_SWAP_SIZE=2g
ZFSBOOT_CONFIRM_LAYOUT=1

The installer would never recognise the second disk and the script would get stuck.

I'm aware that mfsbsd has an option to use a custom root-on-zfs script, but I wanted to use the 'official' FreeBSD tools.

Creating configuration backups of HP procurve switches

2015-01-12T12:00:00+01:00

I've created a tool called procurve-watch. It creates a backup of the running switch configuration through secure shell (using scp).

It also diffs backed up configurations against older versions, in order to keep track of changes. If you run the script from cron every hour or so, you will be notified by email of any (running) configuration changes.

The tool can backup hundreds of switches in seconds as it is running the configuration copy in parallel.

A tool like Rancid may actually be the best choice for this task, but it didn't work. The latest version of Rancid doesn't support HP Procurve switches (yet) and older versions created backups containing garbled characters.

I've released it on github, check it out and let me know if it works for you and you have suggestions to improve it further.

Configuring, attacking and securing VRRP on Linux

2015-01-02T12:00:00+01:00

The VRRP or Virtual Router Redundancy Protocol helps you create a reliable network by using multiple routers in an active/passive configuration. If the primary router fails, the backup router takes over almost seamlessly.

This is how VRRP works:

Clients connect to a virtual IP-address. It is called virtual because the IP-address is not hard-coded to a particular interface on any of the routers.

If a client asks for the MAC-address that is tied to the virtual IP, the master will respond with its MAC-address. If the master dies, the backup router will notice and start responding to ARP-requests.

Let's take a look at the ARP table on the client to illustrate what is happening.

Master is active:

(10.0.1.140) at 0:c:29:a7:7d:f2 on en0 ifscope [ethernet]
(10.0.1.141) at 0:c:29:a7:7d:f2 on en0 ifscope [ethernet]
(10.0.1.142) at 0:c:29:b2:5b:7c on en0 ifscope [ethernet]

Master has failed and backup has taken over:

(10.0.1.140) at 0:c:29:b2:5b:7c on en0 ifscope [ethernet]
(10.0.1.141) at 0:c:29:a7:7d:f2 on en0 ifscope [ethernet]
(10.0.1.142) at 0:c:29:b2:5b:7c on en0 ifscope [ethernet]

Notice how the MAC-address of the virtual IP (.140) is now that of the backup router.

Configuring VRRP on Linux

configure static IP-addresses on the primary and backup router. Do not configure the virtual IP on any of the interfaces. In my test environment, I used 10.0.1.141 for the master and 10.0.1.142 for the backup router.
Because the virtual IP-address is not configured on any of the interfaces, Linux will not reply to any packets destined for this IP. This behaviour needs to be changed or VRRP will not work. Edit /etc/sysctl.conf and add this line:
```
net.ipv4.ip_nonlocal_bind=1
```
Run this command to active this setting:
```
sysctl -p
```
Install Keepalived
```
apt-get install keepalived
```

Sample configuration of /etc/keepalived/keepalived.conf for MASTER

vrrp_instance VI_1 {
    interface eth0
    state MASTER
    virtual_router_id 51
    priority 101

    authentication {
        auth_type AH
        auth_pass monkey
    }

    virtual_ipaddress {
        10.0.1.140
    }
}

Sample configuration of /etc/keepalived/keepalived.conf for SLAVE

vrrp_instance VI_1 {
    interface eth0
    state BACKUP
    virtual_router_id 51
    priority 100

    authentication {
        auth_type AH
        auth_pass monkey
    }

    virtual_ipaddress {
        10.0.1.140
    }
}

Start keepalived:
```
service keepalived start
```

The only configuration difference regarding keepalived between the master and the standby router is the 'priority' setting. The master server should have a higher priority than the backup router (101 vs. 100).

As there can be multiple VRRP configurations active within the same subnet, it is important that you make sure that you set a unique virtual_router_id.

Please do not forget to set your own password in case you enable authentication.

VRRP failover example

This is what happens if the master is shutdown:

64 bytes from 10.0.1.140: icmp_seq=148 ttl=64 time=0.583 ms
64 bytes from 10.0.1.140: icmp_seq=149 ttl=64 time=0.469 ms
64 bytes from 10.0.1.140: icmp_seq=150 ttl=64 time=0.267 ms
Request timeout for icmp_seq 151
Request timeout for icmp_seq 152
Request timeout for icmp_seq 153
Request timeout for icmp_seq 154
64 bytes from 10.0.1.140: icmp_seq=155 ttl=64 time=0.668 ms
64 bytes from 10.0.1.140: icmp_seq=156 ttl=64 time=0.444 ms
64 bytes from 10.0.1.140: icmp_seq=157 ttl=64 time=0.510 ms

After about five seconds (default) the standby router takes over and starts responding to the virtual IP.

Security

A host within the same subnet could just spoof VRRP packets and disrupt service.

An attack on VRRP is not just theoretical. A tool called Loki allows you to take over the virtual IP-address and become the master router. This will allow you to create a DoS or sniff all traffic.

VRRP security is also discussed in this document from the Loki developers.

According to rfc3768 authentication and security has been deliberately omitted (see section 10 Security Considerations) from newer versions of the VRRP protocol RFC.

The main argument is that any malicious device in a layer 2 network can stage similar attacks focussing on ARP-spoofing and ARP-poisoning so as the fundament is already insecure, why care about VRRP?

I understand the reasoning but I disagree. If you do have a secure Layer 2 environment, VRRP becomes the weakest link. Either you really need to filter out VRRP traffic originating from untrusted ports/devices, or implement security on VRRP itself.

Attacking VRRP with Loki

I have actually used Loki on VRRP and I can confirm it works (at least) as a Denial-of-Service tool.

I used Kali (Formerly known as Back-Track) and installed Loki according to these instructions. Please note the bottom of the page.

What I did on Kali Linux:

apt-get install python-dpkt python-dumbnet
wget http://c0decafe.de/svn/codename_loki/packages/kali-1/pylibpcap_0.6.2-1_amd64.deb
wget http://c0decafe.de/svn/codename_loki/packages/kali-1/loki_0.2.7-1_amd64.deb
dpkg -i pylibpcap_0.6.2-1_amd64.deb
dpkg -i loki_0.2.7-1_amd64.deb

Then just run:

loki.py

This is only an issue if you already protected yourself against ARP- and IP-spoofing attacks.

Protecting VRRP against attacks

Keepalived offers two authentication types regarding VRRP:

PASS (plain-text password)
AH (IPSEC-AH (authentication header))

The PASS option is totally useless from a security perspective.

As you can see, the password 'monkey' is visible and easily obtained from the VRRP multicast advertisements. So to me, it does not make sense to use this option. Loki just replayed the packets and could still create a DoS.

So we are left with IPSEC-AH, wich is more promising as it actually does some cryptography using the IPSEC protocol, so there is no clear-text password to be captured. I'm not a crypto expert, so I'm not sure how secure this implementation is. Here is some more info on IPSEC-AH as implemented in Keepalived.

If I configure AH authentication, the Loki tool does not recognise the VRRP trafic anymore and it's no longer possible to use this simple script-kiddie-friendly tool to attack your VRRP setup.

IPSEC-AH actually introduces an IPSEC-AH header between the IP section and the VRRP section of a packet, so it changes the packet format, which probably makes it unrecognisable for Loki.

Running VRRP multicast traffic on different network segments

It has been pointed out to me by XANi_ that it is possible with Keepalived to keep the virtual IP-address and the VRRP multicast traffic in different networks. Clients will therefore not be able to attack the VRRP traffic.

In this case, security on the VRRP traffic is not relevant anymore and you don't really need to worry about authentication, assuming that untrusted devices don't have access to that 'VRRP' VLAN.

Th first step is that both routers should have their physical interface in the same (untagged) VLAN. The trick is then to specify the virtual IP-addresses in the appropriate VLANs like this example:

virtual_ipaddress {

    10.0.1.1/24 dev eth0.100
    10.0.2.1/24 dev eth0.200
}

In this example, virtual IP 10.0.1.1 is tied to VLAN 100 and 10.0.2.1 is tied to VLAN 200.

If the physical router interfaces are present in the untagged VLAN 50 (example), the VRRP multicast traffic will only be observed in this VLAN.

Some background information on working with VLANs and Keepalived.

Firewall configuration

Update August 2018:

I had problems running VRRP on Red Hat / CentOS. Since I use AH authentication, the protocol is not seen as VRRP but (as TCPDUMP shows) "AH". This is why you need to create a service for Firewalld and enable it for the appropriate zone.

Create a file called "VRRP.xml" in /etc/firewalld/services

    <?xml version="1.0" encoding="utf-8"?>
    <service>
        <short>VRRP</short>
        <description>Virtual Router Redundancy Protocol</description>
        <port protocol="ah" port=""/>
    </service>

Enable VRRP (select the appropriate zone for your interface):

    sudo firewall-cmd --zone=public --permanent --add-service=VRRP

Reload the configuration

    sudo firewall-cmd --reload

check that the service is active with:

    sudo firewall-cmd --zone=public --list-services

Closing words

VRRP can provide a very simple solution to setup a high-availability router configuration. Security can be a real issue if untrusted devices reside in the same layer 2 network so implementing security with IPSEC-AH or network segmentation is recommended.

Systemd Forward Secure Sealing of system logs makes little sense

2014-11-22T12:00:00+01:00

Systemd is a more modern replacement of sysvinit and its in the process of being integrated into most mainstream Linux distributions. I'm a bit troubled by one of it's features.

I'd like to discuss the Forward Secure Sealing (FSS) feature for log files that is part of systemd. FSS cryptographically signs the local system logs, so you can check if log files have been altered. This should make it more difficult for an attacker to hide his or her tracks.

Regarding log files, an attacker can do two things:

delete them
alter them (remove / change incriminating lines)

The FSS feature does not prevent any of these risks. But it does help you detect that there is something fishy going on if you would verify the signatures regularly. So basically FSS acts a bit like Tripwire.

FSS can only tell you wether or not a log file has been changed. It cannot tell you anything else. More specifically, it cannot tell you the reason why. So I wonder how valuable this feature is.

There is also something else. Signing (sealing) a log file is done every 15 minutes by default. This gives an attacker ample time to alter or delete the most recent log events, often exactly those events that need to be altered/deleted. Even lowering this number to 10 seconds would allow an attacker to delete (some) initial activities using automation. So how useful is this?

What may help in determining what happened to a system is the unaltered log contents themselves. What FSS cannot do by principle is protect the actual contents of the log file. If you want to preserve log events the only secure option is to send them to an external log host (assumed not accessible by an attacker).

However, to my surprise, FSS is presented as an alternative to external logging. Quote from Lennart Poettering:

Traditionally this problem has been dealt with by having an external secured log server 
to instantly log to, or even a local line printer directly connected to the log system. 
But these solutions are more complex to set up, require external infrastructure and have 
certain scalability problems. With FSS we now have a simple alternative that works without 
any external infrastructure.

This quote is quite troubling because it fails to acknowledge one of the raison d'être of external log hosts. It seems to suggest that FSS provides an alternative for external logging, where in fact it does not and cannot do so on principle. It can never address the fact that an attacker can alter or delete logs, whereas external logging can mitigate this risk.

It seems to me that systemd now also wants to play the role as some crude intrusion detection system. It feels a bit like scope creep to me.

Personally I just wonder what more useful features could have been implemented instead of allowing you to transfer a log file verification key using a QR code to your smartphone (What the hell?).

This whole observation is not original, in the comments of the systemd author's blogpost, the same argument is made by Andrew Wyatt (two years earlier). The response from the systemd author was to block him. (see the comments of Lennart Poettering's blogpost I linked to earlier).

Update: Andrew Wyatt behaved a bit immature towards Lennart Poettering at first so I understand some resentment from his side, but Andrews criticism was valid and never addressed by him.

If the systemd author would just have implemented sending log events to an external log server, that would have been way more useful security-wise, I think. Until then, this may do...

Getting the Sitecom AC600 Wi-Fi adapter running on Linux

2014-11-01T12:00:00+01:00

TL;DR Yes it works with some modifications of the driver source.

A USB Wi-Fi adapter I used with a Raspberry Pi broke as I dropped it on the floor, so I had to replace it. I just went to a local shop and bought the Sitecom AC600 adapter as that's what they had available (with support for 5Ghz networking).

I had some hope that I would just plug it in and it would 'just work™'. But no. Linux. In the end, the device cost me 30 euro's including taxes, but the time spend to get it to work may have made this a very expensive USB Wi-Fi dongle. And it's funny to think about the fact that the Wi-Fi dongle is almost the same price as the Raspberry Pi board itself.

But I did get it working and I'd like to show you how.

It started with a google for 'sitecom ac600 linux' which landed me on this page. This page told me the device uses a MediaTek chipset (MT7610U).

So you need to download the driver from MediaTek. Here is a direct link

So you may do something like this:

cd /usr/src
wget http://s3.amazonaws.com/mtk.cfs/Downloads/linux/mt7610u_wifi_sta_v3002_dpo_20130916.tar.bz2
tar xjf mt7610u_wifi_sta_v3002_dpo_20130916.tar.bz2
cd mt7610u_wifi_sta_v3002_dpo_20130916

Now you would hope that it's just like this:

make
make install

And we're happy right? Linux FTW! Well, NO! We're using Linux so we have to work for stuff that works right out of the box on Windows and Mac OS.

So we first start with editing "include/os/rt_linux.h" and go to line ~279. There we make sure that we edit the struct like this:

    typedef struct _OS_FS_INFO_
 {
    kuid_t              fsuid;
    kgid_t              fsgid;
    mm_segment_t    fs;
 } OS_FS_INFO;

Basically, the words int are replaced by kuid_t and kgid_t, or else, compilation will abort with an error.

Ofcourse, the Sitecom AC600 has an USB identifier that is unknown to the driver, so after compilation, it still doesn't work.

lsusb output:

Bus 001 Device 004: ID 0df6:0075 Sitecom Europe B.V.

So google landed me on this nice thread by 'praseodym' that explained the remaining steps. I stole the info below from this thread.

So while we are in the source directory of the module, we are going to edit "common/rtusb_dev_id.c" and add

{USB_DEVICE(0x0DF6,0x0075)}, /* MT7610U */

So this will make the AC600 gets recognised by the driver. Now we also need to edit "os/linux/confik.mk" and change these lines like this:

HAS_WPA_SUPPLICANT=y
HAS_NATIVE_WPA_SUPPLICANT_SUPPORT=y

So no, we are still not ready yet. I'm not 100 percent sure that this is required anymore, but I found this nice thread in Italian and a very small comment by 'shoe rat' tucked away at the end that may make the difference between a working device or not.

We need to edit the file "os/linux/config.mk" and go to line ~663. Then, around that line, change

CHIPSET_DAT = 2860

to:

CHIPSET_DAT = 2870

Yes. Finally! Now you can do:

make
make install

Imagine that such a 'make' takes about 20 minutes on a Raspbery Pi. No joke.

Now you can either do this:

modprobe mt7650u_sta

You should see something like this:

root@raspberrypi:/usr/src# lsmod
Module                  Size  Used by
snd_bcm2835            16181  0 
snd_pcm                63684  1 snd_bcm2835
snd_page_alloc          3604  1 snd_pcm
snd_seq                43926  0 
snd_seq_device          4981  1 snd_seq
snd_timer              15936  2 snd_pcm,snd_seq
snd                    44915  5 snd_bcm2835,snd_timer,snd_pcm,snd_seq,snd_seq_device
soundcore               4827  1 snd
mt7650u_sta           895786  1 
pl2303                  7951  0 
usbserial              19536  1 pl2303

You should be able to see a 'ra0' device when entering ifconfig -a or iwconfig and just configure it like any wireless device (out-of-scope).

So once up-and-running, the Sitecom AC600 works fine under Linux and even sees and connects to 5 GHz networks. But not without a caveat of-course. I needed to configure a 5 GHz channel below 100 (I chose 48) on my Apple Airport Extreme, or the Wi-Fi dongle would not see the 5GHz network and would not be able to connect to it.

So I hope somebody else is helped by this information.

The ZFS Event Daemon on Linux

2014-08-29T23:00:00+02:00

If something goes wrong with my zpool, I'd like to be notified by email. On Linux using MDADM, the MDADM daemon took care of that.

With the release of ZoL 0.6.3, a brand new 'ZFS Event Daemon' or ZED has been introduced.

I could not find much information about it, so consider this article my notes on this new service.

If you want to receive alerts there is only one requirement: you must setup an MTA on your machine and that is outside the scope of this article.

When you install ZoL, the ZED daemon is installed automatically and will start on boot.

The configuration file for ZED can be found here: /etc/zfs/zed.d/zed.rc. Just uncomment the "ZED_EMAIL=" section and fill out your email address. Don't forget to restart the service.

ZED seems to hook into the zpool event log that is kept in the kernel and monitors these events in real-time.

You can see those events yourself:

root@debian:/etc/zfs/zed.d# zpool events
TIME                           CLASS
Aug 29 2014 16:53:01.872269662 resource.fs.zfs.statechange
Aug 29 2014 16:53:01.873291940 resource.fs.zfs.statechange
Aug 29 2014 16:53:01.962528911 ereport.fs.zfs.config.sync
Aug 29 2014 16:58:40.662619739 ereport.fs.zfs.scrub.start
Aug 29 2014 16:58:40.670865689 ereport.fs.zfs.checksum
Aug 29 2014 16:58:40.671888655 ereport.fs.zfs.checksum
Aug 29 2014 16:58:40.671905612 ereport.fs.zfs.checksum
...

You can see that a scrub was started and that incorrect checksums were discovered. A few seconds later I received an email:

The first email:

A ZFS checksum error has been detected:

  eid: 5
 host: debian
 time: 2014-08-29 16:58:40+0200
 pool: storage
 vdev: disk:/dev/sdc1

And soon thereafter:

A ZFS pool has finished scrubbing:

  eid: 908
 host: debian
 time: 2014-08-29 16:58:51+0200
 pool: storage
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://zfsonlinux.org/msg/ZFS-8000-9P
 scan: scrub repaired 100M in 0h0m with 0 errors on Fri Aug 29 16:58:51 2014
config:

    NAME        STATE     READ WRITE CKSUM
    storage     ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdb     ONLINE       0     0     0
        sdc     ONLINE       0     0   903

errors: No known data errors

Awesome!

The ZED daemon executes commands based on the event class. So it can do more than just send emails, you can customise different actions based on the event class. The event class can be seen in the zpool events output.

One of the more interesting features is automatic replacement of a defect drive with a hot spare, so full fault tolerance is restored as soon as possible.

I've not been able to get this to work. The ZED scripts would not automatically replace a failed/faulted drive.

There seem to be some known issues. The fixes seem to be in a pending pull request.

Just to make sure I got alerted, I've simulated the ZED configuration for my production environment in a VM.

I simulated a drive failure with dd as stated earlier, but the result was that for every checksum error I received one email. With thousands of checksum errors, I had to clear 1000+ emails from my inbox.

It seems that this option, which is uncommented by default, was not enabled.

ZED_EMAIL_INTERVAL_SECS="3600"

This option implements a cool-down period where an event is just reported once and suppressed afterwards until the interval expires.

It would be best if this option would be enabled by default.

The ZED authors acknowledge that ZED is a bit rough around the edges, but it sends out alerts consistently and that's what I was looking for, so I'm happy.

Installation of ZFS on Linux hangs on Debian Wheezy

2014-08-29T12:00:00+02:00

This article is no longer relevant.

After a fresh net-install of Debian Wheezy, I was unable to compile the ZFS for Linux kernel module. I've installed apt-get install build-essential but that wasn't enough.

The apt-get install debian-zfs command would just hang.

I noticed a 'configure' process and I killed it, and after a few seconds, the installer continued after spewing out this error:

Building initial module for 3.2.0-4-amd64
Error! Bad return status for module build on kernel: 3.2.0-4-amd64 (x86_64)
Consult /var/lib/dkms/zfs/0.6.3/build/make.log for more information.

So I ran ./configure manually inside the mentioned directory and then I got this error:

checking for zlib.h... no
configure: error: in `/var/lib/dkms/zfs/0.6.3/build':
configure: error: 
    *** zlib.h missing, zlib-devel package required
See `config.log' for more details

So I ran apt-get install zlib1g-dev and no luck:

checking for uuid/uuid.h... no
configure: error: in `/var/lib/dkms/zfs/0.6.3/build':
configure: error: 
    *** uuid/uuid.h missing, libuuid-devel package required
See `config.log' for more details

I searched a bit online and then I found this link that listed some additional packages that may be missing and I installed them all with:

apt-get install zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted
lsscsi wget

This time the ./configure went fine and I could manually make install the kernel module and import my existing pool.

Please use ZFS with ECC Memory

2014-08-27T12:00:00+02:00

In this blogpost I argue why it's strongly recommended to use ZFS with ECC memory when building a NAS. I would argue that if you do not use ECC memory, it's reasonable to also forgo on ZFS altogether and use any (legacy) file system that suits your needs.

Why ZFS?

Many people consider using ZFS when they are planning to build their own NAS. This is for good reason: ZFS is an excellent choice for a NAS file system. There are many reasons why ZFS is such a fine choice, but the most important one is probably 'data integrity'. Data integrity was one of the primary design goals of ZFS.

ZFS assures that any corrupt data served by the underlying storage system is either detected or - if possible - corrected by using checksums and parity. This is why ZFS is so interesting for NAS builders: it's OK to use inexpensive (consumer) hard drives and solid state drives and not worry about data integrity.

I will not go into the details, but for completeness I will also state that ZFS can make the difference between losing an entire RAID array or just a few files, because of the way it handles read errors as compared to 'legacy' hardware/software RAID solutions.

Understanding ECC memory

ECC memory or Error Correcting Code memory, contains extra parity data so the integrity of the data in memory can be verified and even corrected. ECC memory can correct single bit errors and detect multiple bit errors per word¹.

What's most interesting is how a system with ECC memory reacts to bit errors that cannot be corrected. Because it's how a system with ECC memory responds to uncorrectable bit errors that that makes all the difference in the world.

If multiple bits are corrupted within a single word, the CPU will detect the errors, but will not be able to correct them. When the CPU notices that there are uncorrectable bit errors in memory, it will generate an MCE that will be handled by the operating system. In most cases, this will result in a halt² of the system.

This behaviour will lead to a system crash, but it prevents data corruption. It prevents the bad bits from being processed by the operating system and/or applications where it may wreak havoc.

ECC memory is standard on all server hardware sold by all major vendors like HP, Dell, IBM, Supermicro and so on. This is for good reason, because memory errors are the norm, not the exception.

The question is really why not all computers, including desktop and laptops, use ECC memory instead of non-ECC memory. The most important reason seems to be 'cost'.

It is more expensive to use ECC memory than non-ECC memory. This is not only because ECC memory itself is more expensive. ECC memory requires a motherboard with support for ECC memory, and these motherboards tend to be more expensive as well.

non-ECC Memory is reliable enough that you won't have an issue most of the time. And when it does go wrong, you just blame Microsoft or Apple³. For desktops, the impact of a memory failure is less of an issue than on servers. But remember, your NAS is your own (home) server. There is some evidence that memory errors are abundant⁴ on desktop systems.

The price difference is small enough not to be relevant for businesses, but for the price-conscious consumer, it is a factor. A system based on ECC memory may cost in the range of $150 - $200 more than a system based on non-ECC memory.

It's up to you if you want to spend this extra money. Why you are advised to do so will be discussed in the next paragraphs.

Why ECC memory is important to ZFS

ZFS trusts the contents of memory blindly. Please note that ZFS has no mechanisms to cope with bad memory. It is similar to every other file system in this regard. Here is a nice paper about ZFS and how it handles corrupt memory (it doesnt!).

In the best case, bad memory corrupts file data and causes a few garbled files. In the worst case, bad memory mangles in-memory ZFS file system (meta) data structures, which may lead to corruption and thus loss of the entire zpool.

It is important to put this into perspective. There is only a practical reason why ECC memory is more important for ZFS as compared to other file systems. Conceptually, ZFS does not require ECC memory any more as any other file system.

Or let Matthew Ahrens, the co-founder of the ZFS project phrase it:

There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than 
any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS.

Now this is the important part. File systems such as NTFS, EXT4, etc have (data recovery) tools that may allow you to rescue your files when things go bad due to bad memory. ZFS does not have such tools, if the pool is corrupt, all data must be considered lost, there is no option for recovery.

So the impact of bad memory can be more devastating on a system with ZFS than on a system with NTFS, EXT4, XFS, etcetera. ZFS may force you to restore your data from backups sooner. Oh by the way, you, make backups right?

I do have a personal concern⁵. I have nothing to substantiate this, but my thinking is that since ZFS is a way more advanced and complex file system, it may be more susceptible to the adverse effects of bad memory, compared to legacy file systems.

ZFS, ECC memory and data integrity

The main reason for using ZFS over legacy file systems is the ability to assure data integrity. But ZFS is only one piece of the data integrity puzzle. The other part of the puzzle is ECC memory.

ZFS covers the risk of your storage subsystem serving corrupt data. ECC memory covers the risk of corrupt memory. If you leave any of these parts out, you are compromising data integrity.

If you care about data integrity, you need to use ZFS in combination with ECC memory. If you don't care that much about data integrity, it doesn't really matter if you use either ZFS or ECC memory.

Please remember that ZFS was developed to assure data integrity in a corporate IT environment, where data integrity is top priority and ECC-memory in servers is the norm, a fundament, on wich ZFS has been build. ZFS is not some magic pixie dust that protects your data under all circumstances. If its requirements are not met, data integrity is not assured.

ZFS may be free, but data integrity and availability isn't. We spend money on extra hard drives so we can run RAID(Z) and lose one or more hard drives without losing our data. And we have to spend money on ECC-memory, to assure bad memory doesn't have a similar impact.

This is a bit of an appeal to authority and not to data or reason but I think it's still relevant. FreeNAS is a vendor of a NAS solution that uses ZFS as its foundation.

They have this to say about ECC memory:

However if a non-ECC memory module goes haywire, it can cause irreparable damage to your ZFS pool that can cause complete loss of the storage.
...
If it’s imperative that your ZFS based system must always be available, ECC RAM is a requirement. If it’s only some level of annoying (slightly, moderately…) that you need to restore 
your ZFS system from backups, non-ECC RAM will fit the bill.

Hopefully your backups won't contain corrupt data. If you make backups of all data in the first place.

Many home NAS builders won't be able to afford to backup all data on their NAS, only the most critical data. For example, if you store a large collection of video files, you may accept the risk that you may have to redownload everything. If you can't accept that risk ECC memory is a must. If you are OK with such a scenario, non-ECC memory is OK and you can save a few bucks. It all depends on your needs.

The risks faced in a business environment don't magically disapear when you apply the same technology at home. The main difference between a business setting and your home is the scale of operation, nothing else. The risks are still relevant and real.

Things break, it's that simple. And although you may not face the same chances of getting affected by it based on the smaller scale at which you operate at home, your NAS is probably not placed in a temperature and humidity controlled server room. As the temperature rises, so does the risk of memory errors⁶. And remember, memory may develop spontaneous and temporary defects (random bitflips). If your system is powered on 24/7, there is a higher chance that such a thing will happen.

Conclusion

Personally, I think that even for a home NAS, it's best to use ECC memory regardless if you use ZFS. It makes for a more stable hardware platform. If money is a real constraint, it's better to take a look at AMD's offerings then to skip on ECC memory. It's important that if you select AMD hardware, that you make sure that both CPU and motherboard support ECC and that it is reported to be working.

Still, if you decide to use non-ECC memory with ZFS: as long as you are aware of the risks outlined in this blog post and you're OK with that, fine. It's your data and you must decide for yourself what kind of protection and associated cost is reasonable for you.

When people seek advice on their NAS builds, ECC memory should always be recommended. I think that nobody should create the impression that it's 'safe' for home use not to use ECC RAM purely seen from a technical and data integrity standpoint. People must understand that they are taking a risk. But there is a significant chance that they will never experience problems, but there is no guarantee. Do they accept the consequences if it does go wrong?

If data integrity is not that important - because the data itself is not critical - I find it perfectly reasonable that people may decide not to use ECC memory and save a few hundred dollars. In that case, it would also be perfectly reasonable not to use ZFS either, which also may allow them other file system and RAID options that may better suit their particular needs.

Questions and answers

Q: When I bought my non-ECC memory, I ran memtest86+ and no errors were found, even after a burn-in tests. So I think I'm safe.

A: No. A memory test with memtest86+ is just a snapshot in time. At that time, when you ran the test, you had the assurance that memory was fine. It could have gone bad right now while you are reading these words. And could be corrupting your data as we speak. So running memtest86+ frequently doesn't really buy you much.

Q: Dit you see that article by Brian Moses?

A: yes, and I disagree with his views, but I really appreciate the fact that he emphasises that you should really be aware of the risks involved and decide for yourself what suits your situation. A few points that are not OK in my opinion:

Every bad stick of RAM I’ve experienced came to me that way from the factory and could be found via some burn-in testing.

I've seen some consumer equipment in my life time that suddenly developed memory errors after years of perfect operation. This is argument from personal anekdote should not be used as a basis for decision making. Remember: memory errors are the norm, not the exception. Even at home. Things break, it's that simple. And having equipment running 24/7 doesn't help.

Furthermore, Brian seems to think that you can mitigate the risk of non-ECC memory by spending money on other stuff, such as off-site backups. Brian himself links to an article that rebutes his position on this. Just for completeness: How valuable is a backup of corrupted data? How do you know which data was corrupted? ZFS won't save you here.

Q: Should I use ZFS on my laptop or desktop?

A: Running ZFS on your desktop or laptop is an entirely different use case as compared to a NAS. I see no problems with this, I don't think this discussion applies to desktop/laptop usage. Especially because you are probably creating regular backups of your data to your NAS or a cloud service, right? If there are any memory errors, you will notice soon enough.

Updates

Updated on August 11, 2015 to reflect that ZFS was not designed with ECC in mind. In this regard, it doesn't differ from other file systems.
Updated on April 3rd, 2015 - rewrote large parts of the whole article, to make it a better read.
Updated on January 18th, 2015 - rephrased some sentences. Changed the paragraph 'Inform people and give them a choice' to argue when it would be reasonable not to use ECC memory. Furthermore, I state more explicitly that ZFS itself has no mechanisms to cope with bad RAM.
Updated on February 21th, 2015 - I substantially rewrote this article to give a better perspective on the ZFS + ECC 'debate'.

On x64 processors, the size of a word is 64 bits. ↩
Windows will generate a "blue screen of death" and Linux will generate a "kernel panic". ↩
It is very likely that the computer you're using (laptop/desktop) encountered a memory issue this year, but there is no way you can tell. Consumer hardware doesn't have any mechanisms to detect and report memory errors. ↩
Microsoft has performed a study on one million crash reports they received over a period of 8 months on roughly a million systems in 2008. The result is a 1 in 1700 failure rate for single-bit memory errors in kernel code pages (a tiny subset of total memory).

:::text A consequence of confining our analysis to kernel code pages is that we will miss DRAM failures in the vast majority of memory. On a typical machine kernel code pages occupy roughly 30 MB of memory, which is 1.5% of the memory on the average system in our study. [...] since we are capturing DRAM errors in only 1.5% of the address space, it is possible that DRAM error rates across all of DRAM may be far higher than what we have observed. ↩
I did not come up with this argument myself. ↩
The absolutely facinating concept of bitsquatting proved that hotter datacenters showed more bitflips ↩

71 TiB DIY NAS based on ZFS on Linux

2014-08-02T21:08:00+02:00

This is my new 71 TiB DIY NAS. This server is the successor to my six year old, twenty drive 18 TB NAS (17 TiB). With a storage capacity four times higher than the original and an incredible read (2.5 GB/s)/write (1.9 GB/s) performance, it's a worthy successor.

Purpose

The purpose of this machine is to store backups and media, primarily video.

The specs

Part	Description
Case	Ri-vier RV-4324-01A
Processor	Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
RAM	16 GB
Motherboard	Supermicro X9SCM-F
LAN	Intel Gigabit
Storage Connectivity	~~InfiniBand MHGA28-XTC~~ 2023: Mellanox ConnectX-3 Pro 10Gbit Ethernet (X312B-XCCT)
PSU	Seasonic Platinum 860
Controller	3 x IBM M1015
Disk	24 x HGST HDS724040ALE640 4 TB (7200RPM)
SSD	2 x Crucial M500 120GB
Arrays	Boot: 2 x 60 GB RAID 1 and storage: 18 disk RAIDZ2+ 6 disk RAIDZ2
Brutto storage	86 TiB (96 TB)
Netto storage	71 TiB (78 TB)
OS	~~Linux Debian Wheezy~~2023: Ubuntu 22.04
Filesystem	ZFS
Rebuild time	Depends on amount of data (rate is 4 TB/Hour)
UPS	Back-UPS RS 1200 LCD using Apcupsd
Power usage	about 200 Watt idle

CPU

The Intel Xeon E3-1230 V2 is not the latest generation but one of the cheapest Xeons you can buy and it supports ECC memory. It's a quad-core processor with hyper-threading.

Here you can see how it performs compared to other processors.

Memory

The system has 16 GB ECC RAM. Memory is relatively cheap these days but I don't have any reason to upgrade to 32 GB. I think that 8 GB would have been fine with this system.

Motherboard

The server is build around the SuperMicro X95SCM-F motherboard.

This is a server-grade motherboard and comes with typical features you might expect, like ECC memory support and out-of-band management (IPMI).

This motherboard has four PCIe slots (2 x 8x and 2 x 4x) in an 8x physical slot. My build requires four PCIe 4x+ slots and there aren't (m)any other server boards at this price point that support four PCIe slots in a 8x sized slot.

The chassis

The chassis has six rows of four drive bays that are kept cool by three 120mm fans in a fan wall behind the drive bays. At the rear of the case, there are two 'powerful' 80mm fans that remove the heat from the case, together with the PSU.

The chassis has six SAS backplanes that connect four drives each. The backplanes have dual molex power connectors, so you can put redundant power supplies into the chassis. Redundant power supplies are more expensive and due to their size, often have smaller, thus noisier fans. As this is a home build, I opted for just a single regular PSU.

When facing the front, there is a place at the left side of the chassis to mount a single 3.5 inch or two 2.5 inch drives next to each other as boot drives. I've mounted two SSDs (RAID1).

This particular chassis version has support for SPGIO, which should help identifying which drive has failed. The IBM 1015 cards I use do support SGPIO. Through the LSI megaraid CLI I have verified that SGPIO works, as you can use this tool as a drive locator. I'm not entirely sure how well SGPIO works with ZFS.

Power supply

I was using a Corsair 860i before, but it was unstable and died on me.

The Seasonic Platinum 860 may seem like overkill for this system. However, I'm not using staggered spinup for the 24 drives. So the drives all spinup at once and this results in a peak power usage of 600+ watts.

The PSU has a silent mode that causes the fan only to spin if the load reaches a certain threshold. Since the PSU fan also helps removing warm air from the chassis, I've disabled this feature, so the fan is spinning at all times.

Drive management

I've written a tool called lsidrivemap that displays each drive in an ASCII table that reflects the physical layout of the chassis.

The data is based on the output of the LSI 'megacli' tool for my IBM 1015 controllers.

root@nano:~# lsidrivemap disk

| sdr | sds | sdt | sdq |
| sdu | sdv | sdx | sdw |
| sdi | sdl | sdp | sdm |
| sdj | sdk | sdn | sdo |
| sdb | sdc | sde | sdf |
| sda | sdd | sdh | sdg |

This layout is 'hardcoded' for my chassis but the Python script can be easily tailored for your own server, if you're interested.

It can also show the temperature of the disk drives in the same table:

root@nano:~# lsidrivemap temp

| 36 | 39 | 40 | 38 |
| 36 | 36 | 37 | 36 |
| 35 | 38 | 36 | 36 |
| 35 | 37 | 36 | 35 |
| 35 | 36 | 36 | 35 |
| 34 | 35 | 36 | 35 |

These temperatures show that the top drives run a bit hotter than the other drives. An unverified explanation could be that the three 120mm fans are not in the center of the fan wall. They are skewed to the bottom of the wall, so they may favor the lower drive bays.

Filesystem (ZFS)

I'm using ZFS as the file system for the storage array. At this moment, there is no other file system that has the same features and stability as ZFS. BTRFS is not even finished.

The number one design goal of ZFS was assuring data integrity. ZFS checksums all data and if you use RAIDZ or a mirror, it can even repair data. Even if it can't repair a file, it can at least tell you which files are corrupt.

ZFS is not primarily focussed on performance, but to get the best performance possible, it makes heavy usage of RAM to cache both reads and writes. This is why ECC memory is so important.

ZFS also implements RAID. So there is no need to use MDADM. My previous file server was running a single RAID 6 of 20 x 1TB drives. With this new system I've created a single pool with two RAIDZ2 VDEVs.

Capacity

Vendors still advertise the capacity of their hard drives in TB whereas the operating system works with TiB. So the 4 TB drives I use are in fact 3.64 TiB.

The total raw storage capacity of the system is about 86 TiB.

My zpool is the 'appropriate' number of disks (2^n + parity^) in the VDEVs. So I have one 18 disk RAIDZ2 VDEV (2^4+2) and one 6 disk RAIDZ2 VDEV (2^2+2^) for a total of 24 drives.

Different VDEV sizes in a single pool are often not recommended, but ZFS is very smart and cool: it load-balances the data across the VDEVs based on the size of the VDEV. I could verify this with zpool iostat -v 5 and witness this in real-time. The small VDEV got just a fraction of the data compared to the large VDEV.

This choice leaves me with less capacity (71 TiB vs. 74 TiB for RAIDZ3) and also has a bit more risk to it, with the eighteen-disk RAIDZ2 VDEV. Regarding this latter risk, I've been running a twenty-disk MDADM RAID6 for the last 6 years and haven't seen any issues. That does not tell everything, but I'm comfortable with this risk.

Originalyl I was planning on using RAIDZ3 and by using ashift=9 (512 byte sectors) I would recuperate most of the space lost to the non-optimal number of drives in the VDEV. So why did I change my mind? Because the performance of my ashift=9 pool on my 4K drives deteriorated so much that a resilver of a failed drive would take ages.

Storage controllers

The IBM 1015 HBA's are reasonably priced and buying three of them, is often cheaper than buying just one HBA with a SAS expander. However, it may be cheaper to search for an HP SAS expander and use it with just one M1015 and save a PCIe slot.

I have not flashed the controllers to 'IT mode', as most people do. They worked out-of-the-box as HBAs and although it may take a little bit longer to boot the system, I decided not to go through the hassle.

The main risk here is how the controller handles a drive if a sector is not properly read. It may disable the drive entirely, which is not necessary for ZFS and often not preferred.

Storage performance

With twenty-four drives in a chassis, it's interesting to see what kind of performance you can get from the system.

Let's start with a twenty-four drive RAID 0. The drives I use have a sustained read/write speed of 160 MB/s so it should be possible to reach 3840 MB/s or 3.8 GB/s. That would be amazing.

This is the performance of a RAID 0 (MDADM) of all twenty-four drives.

root@nano:/storage# dd if=/dev/zero of=test.bin bs=1M count=1000000
1048576000000 bytes (1.0 TB) copied, 397.325 s, 2.6 GB/s

root@nano:/storage# dd if=test.bin of=/dev/null bs=1M
1048576000000 bytes (1.0 TB) copied, 276.869 s, 3.8 GB/s

Dead on, you would say, but if you divide 1 TB with 276 seconds, it's more like 3.6 GB/s. I would say that's still quite close.

This machine will be used as a file server and a bit of redundancy would be nice. So what happens if we run the same benchmark on a RAID6 of all drives?

root@nano:/storage# dd if=/dev/zero of=test.bin bs=1M count=100000
104857600000 bytes (105 GB) copied, 66.3935 s, 1.6 GB/s

root@nano:/storage# dd if=test.bin of=/dev/null bs=1M
104857600000 bytes (105 GB) copied, 38.256 s, 2.7 GB/s

I'm quite pleased with these results, especially for a RAID6. However, RAID6 with twenty-four drives feels a bit risky. So since there is no support for a three-parity disk RAID in MDADM/Linux, I use ZFS.

Sacrificing performance, I decided - as I mentioned earlier - to use ashift=9 on those 4K sector drives, because I gained about 5 TiB of storage in exchange.

This is the performance of twenty-four drives in a RAIDZ3 VDEV with ashift=9.

root@nano:/storage# dd if=/dev/zero of=ashift9.bin bs=1M count=100000 
104857600000 bytes (105 GB) copied, 97.4231 s, 1.1 GB/s

root@nano:/storage# dd if=ashift9.bin of=/dev/null bs=1M
104857600000 bytes (105 GB) copied, 42.3805 s, 2.5 GB/s

Compared to the other results, write performance is way down, although not too bad.

This is the write performance of the 18 disk RAIDZ2 + 6 disk RAIDZ2 zpool (ashift=12):

root@nano:/storage# dd if=/dev/zero of=test.bin bs=1M count=1000000 
1048576000000 bytes (1.0 TB) copied, 543.072 s, 1.9 GB/s

root@nano:/storage# dd if=test.bin of=/dev/null bs=1M 
1048576000000 bytes (1.0 TB) copied, 400.539 s, 2.6 GB/s

As you may notice, the write performance is better than the ashift=9 or ashift=12 RAIDZ3 VDEV.

In the end I chose to use the 18 disk RAIDZ2 + 6 disk RAIDZ2 setup because of the better performance and to adhere to the standards of ZFS.

I have not benchmarked random I/O performance as it is not relevant for this system. And with ZFS, the random I/O performance of a VDEV is that of a single drive.

Boot drives

I'm using two Crucial M500 120GB SSD drives. They are configured in a RAID1 (MDADM) and I've installed Debian Wheezy on top of them.

At first, I was planning on using a part of the capacity for caching purposes in combination with ZFS. However, there's no real need to do so. In hindsight I could also have used to very cheap 2.5" hard drives (simmilar to my older NAS), which would have cost less than a single M500.

Update 2014-09-01: I actually reinstalled Debian and kept about 50% free space on both M500s and put this space in a partition. These partitions have been provided to the ZFS pool as L2ARC cache. I did this because I could, but on the other hand, I wonder if I'm only really just wearing out my SSDs faster.

Update 2015-10-04: I saw no reason why I would wear out my SSDs as a L2ARC so I removed them from my pool. There is absolutely no benefit in my case.

Networking (updated 2017-03-25)

Current: I have installed a Mellanox MHGA28-XTC InfiniBand card. I'm using InfiniBand over IP so the InfiniBand card is effectively a faster network card. I have a point-to-point connection with another server, I do not have an InfiniBand switch.

I get about 6.5 Gbit from this card, which is not even near the theoretical performance limit. However, this translate into a constant 750 MB/s file transfer speed over NFS, which is amazing.

Using Linux bonding and the quad-port Ethernet adapter, I only got 400 MB/s and transfer speeds were fluctuating a lot.

Original: Maybe I will invest in 10Gbit ethernet or InfiniBand hardware in the future, but for now I settled on a quad-port gigabit adapter. With Linux bonding, I can still get 450+ MB/s data transfers, which is sufficient for my needs.

The quad-port card is in addition to the two on-board gigabit network cards. I use one of the on-board ports for client access. The four ports on the quad-port card are all in different VLANs and not accessible for client devices.

The storage will be accessible over NFS and SMB. Clients will access storage over one of the on-board Gigabit LAN interfaces.

Keeping things cool and quiet

It's important to keep the drive temperature at acceptable levels and with 24 drives packet together, there is an increased risk of overheating.

The chassis is well-equipped to keep the drives cool with three 120mm fans and two strong 80mm fans, all supporting PWM (pulse-width modulation).

The problem is that by default, the BIOS runs the fans at a too low speed to keep the drives at a reasonable temperature. I'd like to keep the hottest drive at about forty degrees Celsius. But I also want to keep the noise at reasonable levels.

I wrote a python script called storagefancontrol that automatically adjusts the fan speed based on the temperature of the hottest drive.

UPS

I'm running a HP N40L micro server as my firewall/router. My APC Back-UPS RS 1200 LCD (720 Watt) is connected with USB to this machine. I'm using apcupsd to monitor the UPS and shutdown servers if the battery runs low.

All servers, including my new build, run apcupsd in network mode and talk to the N40L to learn if power is still OK.

Keeping power consumption reasonable

So these are the power usage numbers.

 96 Watt with disks in spin down.
176 Watt with disks spinning but idle.
253 Watt with disks writing.

Edit 2015-10-04: I do have an unresolved issue where the drives keep spinning up even with all services on the box killed, including Cron. So it's configured so that the drives are always spinning. /end edit

But the most important stat is that it's using 0 Watt if powered off. The system will be turned on only when necessary through wake-on-lan. It will be powered off most of the time, like when I'm at work or sleeping.

Cost

The system has cost me about €6000. All costs below are in Euro and include taxes (21%).

Description	Product	Price	Amount	Total
Chassis	Ri-vier 4U 24bay storage chassis RV-4324-01A	554	1	554
CPU	Intel Xeon E3-1230V2	197	1	197
Mobo	SuperMicro X9SCM-F	157	1	157
RAM	Kingston DDR3 ECC KVR1333D3E9SK2/16G	152	1	152
PSU	AX860i 80Plus Platinum	175	1	175
Network Card	NC364T PCI Express Quad Port Gigabit	145	1	145
HBA Controller	IBM SERVERAID M1015	118	3	354
SSDs	Crucial M500 120GB	62	2	124
Fan	Zalman FB123 Casefan Bracket + 92mm Fan	7	1	7
Hard Drive	Hitachi 3.5 4TB 7200RPM (0S03356)	166	24	3984
SAS Cables		25	6	150
Fan cables		6	1	6
Sata-to-Molex		3,5	1	3,5
Molex splitter		3	1	3
				6012

Closing words

If you have any questions or remarks about what could have been done differently feel free to leave a comment, I appreciate it.

ZFS: Performance and capacity impact of ashift=9 on 4K sector drives

2014-07-31T00:00:00+02:00

Update 2014-8-23: I was testing with ashift for my new NAS. The ashift=9 write performance deteriorated from 1.1 GB/s to 830 MB/s with just 16 TB of data on the pool. Also I noticed that resilvering was very slow. This is why I decided to abandon my 24 drive RAIDZ3 configuration.

I'm aware that drives are faster at the outside of the platter and slower on the inside, but the performance deteriorated so dramatically that I did not wanted to continue further.

My final setup will be a RAIDZ2 18 drive VDEV + RAIDZ2 6 drive VDEV which will give me 'only' 71 TiB of storage, but read performance is 2.6 GB/s and write performance is excellent at 1.9 GB/s. I've written about 40+ TiB to the array and after those 40 TiB, write performance was about 1.7 GB/s, so still very good and what I would expect as drives fill up.

So actually, based on these results, I have learned not to deviate from the ZFS best practices too much. Use ashift=12 and put drives in VDEVS that adhere to the 2^n+parity rule.

The uneven VDEVs (18 disk vs. 6 disks) are not according to best practice but ZFS is smart: it distributes data across the VDEVs based on their size. So they fill up equally.

Choosing between ashift=9 and ashift=12 for 4K sector drives is not always a clear cut case. You have to choose between raw performance or storage capacity.

My testplatform is Debian Wheezy with ZFS on Linux. I'm using a system with 24 x 4 TB drives in a RAIDZ3. The drives have a native sector size of 4K, and the array is formatted with ashift=12.

First we create the array like this:

zpool create storage -o ashift=12 raidz3 /dev/sd[abcdefghijklmnopqrstuvwx]

Note: NEVER use /dev/sd? drive names for an array, this is just for testing, always use /dev/disk/by-id/ names.

Then we run a simple sequential transfer benchmark with dd:

root@nano:/storage# dd if=/dev/zero of=ashift12.bin bs=1M count=100000 
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 66.4922 s, 1.6 GB/s
root@nano:/storage# dd if=ashift12.bin of=/dev/null bs=1M
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 42.0371 s, 2.5 GB/s

This is quite impressive. With these speeds, you can saturate 10Gbe ethernet. But how much storage space do we get?

df -h:

Filesystem                            Size  Used Avail Use% Mounted on
storage                                69T  512K   69T   1% /storage

zfs list:

NAME      USED  AVAIL  REFER  MOUNTPOINT
storage  1.66M  68.4T   435K  /storage

Only 68.4 TiB of storage? That's not good. There should be 24 drives minus 3 for parity is 21 x 3.6 TiB = 75 TiB of storage.

So the performance is great, but somehow, we lost about 6 TiB of storage, more than a whole drive.

So what happens if you create the same array with ashift=9?

zpool create storage -o ashift=9 raidz3 /dev/sd[abcdefghijklmnopqrstuvwx]

These are the benchmarks:

root@nano:/storage# dd if=/dev/zero of=ashift9.bin bs=1M count=100000 
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 97.4231 s, 1.1 GB/s
root@nano:/storage# dd if=ashift9.bin of=/dev/null bs=1M
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 42.3805 s, 2.5 GB/s

So we lose about a third of our write performance, but the read performance is not affected, probably by read-ahead caching but I'm not sure.

With ashift=9, we do lose some write performance, but we can still saturate 10Gbe.

Now look what happens to the available storage capacity:

df -h:

Filesystem                         Size  Used Avail Use% Mounted on
storage                             74T   98G   74T   1% /storage

zfs list:

NAME      USED  AVAIL  REFER  MOUNTPOINT
storage   271K  73.9T  89.8K  /storage

Now we have a capacity of 74 TiB, so we just gained 5 TiB with ashift=9 over ashift=12, at the cost of some write performance.

So if you really care about sequential write performance, ashift=12 is the better option. If storage capacity is more important, ashift=9 seems to be the best solution for 4K drives.

The performance of ashift=9 on 4K drives is always described as 'horrible' but I think it's best to run your own benchmarks and decide for yourself.

Caveat: I'm quite sure about the benchmark performance. I'm not 100% sure how reliable the reported free space is according to df -h or zfs list.

Edit: I have added a bit of my own opinion on the results.

Achieving 2.3 GB/s with 16 x 4 TB drives

2014-07-12T12:00:00+02:00

I'm in the process of building a new storage server to replace my 18 TB NAS.

The server is almost finished, it's now down to adding disk drives. I'm using the HGST 4 TB 7200 RPM drive for this build (SKU 0S03356) (review).

I have not bought all drives at once, but slowly adding them in smaller quantities. I just don't want to feel too much pain in my wallet at once I guess.

According to my own tests, this drive has a read/write throughput of 160 MB/s, which is in tune with it's specification.

So the theoretical performance of a RAID 0 with 16 drives x 160 MB/s = 2560 MB/s. That's over 2.5 gigabytes per second.

This is the actual real-life performance I was able to achieve.

root@nano:/storage# dd if=pureawesomeness.dd of=/dev/null bs=1M
1000000+0 records in
1000000+0 records out
1048576000000 bytes (1.0 TB) copied, 453.155 s, 2.3 GB/s

Personally, 2.3 GB/s is not too shabby in my opinion. Please note that I used a test file of one terabyte, so the 16 GB of RAM my server has, doesn't skew the result.

This result is very nice, but in practice almost useless. I can saturate dual 10 Gbit NICs with this system, but I don't have that kind of equipment or any other device that could handle such performance.

But I think it's amazing anyway.

I'm quite curious how the final 24 drive array will perform in a RAID 0.

Affordable server with server-grade hardware part II

2014-06-20T12:00:00+02:00

If you want to build a home server, it may be advised to actually use server-grade components. I documented the reasons for choosing server-grade hardware already in an earlier post on this topic.

It is recommended to read the old post first. In this new post, I only show new hardware that could also be chosen as a more modern hardware option.

My original post dates back to December 2013 and centers around the popular X9SCM-F which is based on the LGA 1155 socket. Please note that the X9SCM-F / LGA 1155 based solution may be cheaper if you want the Xeon processor.

So I'd like to introduce two Supermicro motherboards that may be of interest.

Supermicro X10SLL-F

Some key features are:

2 x Gigabit NIC on-board
6 onboard SATA ports
3 x PCIe (2 x 8x + 1 x 4x)
Costs $169 or €160

This board is one of the cheapest Supermicro boards you can get and it has 3 x PCI-e, which may be of interest if you need to install extra HBA's or RAID cards, SAS expanders and/or network controllers.

Supermicro X10SL7-F

This board is about $80 or €90 more expensive than the X10SLL-F but in return, you get eight extra SAS/SATA ports, for a total of 14 SATA ports. With 4 TB drives, this would give you 56 TB of raw storage capacity. This motherboard provides a cheaper solution than an add-on HBA card, which would occupy a PCIe slot. Hoever, the's a caveat: this board has 'only' two PCIe slots. But there's still room for an additional quad-port or 10 Gbe NIC and an extra HBA if required.

2 x Gigabit NIC on-board
6 onboard SATA ports
8 onboard SAS/SATA ports via LSI 2308 chip
2 x PCIe (8x and 4x)
Costs $242 or €250

Overview of CPU's

CPU	Passmark score	Price in Euro	Price in Dollars
Intel Pentium G3420 @ 3.20GHz	3459	55 Euro	74 Dollar
Intel Core i3-4130 @ 3.40GHz	4827	94 Euro	124 Dollar
Intel Xeon E3-1230 V3 @ 3.30GHz	9459	216Euro	279 Dollar

Dollars are from Newegg, Euro's are from Tweakers.net.
Euros are including taxes.

How to resolve extreme memory usage on Windows 2008 R2-based file servers

2014-06-15T12:00:00+02:00

I'm responsible for a file server with about 5 terrabytes of data. The file server is based on Windows 2008 R2. I've noticed extreme memory usage on the server. After a reboot, it slowly builds up until almost all RAM memory is consumed.

So I googled around and found this post and it turned out I had the same exact issue.

I've confirmed with the tool 'RAMmap' that NTFS metadata is the issue. Microsoft also created a blog post about this.

The author of the first article resolved the issue by adding more RAM memory. But with 16 GB already assigned, I was not to happy to add more memory to the virtual file server, eating away on the RAM resources of our virtualisation platform.

I could never find a root cause of the issue. In that case, you need to obtain the 'Microsoft Windows Dynamic Cache Service'. This application allows you to configure how large the medata caching may grow.

Please note that this services is not a next-next-finish installation. Follow the included Word document with instructions carefully and configure a sane memory setting for your server. I limited the cache to half the RAM available to the server and this works out well.

My experiences with DFS replication on Windows 2008 R2

2014-06-15T12:00:00+02:00

If you are considering implementing DFS replication, consider using Windows 2012 R2 because DFS replication has been massively improved. It supports larger data sets and performance has dramatically been improved over Windows 2008 R2.

I've implemented DFS replication to keep two file servers synchronised. Click here if or there you want to learn more about DFS itself.

With DFS, I wanted to create a high-available file server service, based on two file servers, each with their own physical storage. DFS replication makes sure that both file servers are kept in sync.

If you setup DFS, you need to copy all the data from the original server to the secondary server. This is called seeding and I've used robocopy as recommended by Microsoft in the linked article.

Seeding is not mandatory. You can just start with an empty folder on the secondary server and just have DFS replicate all files. I've experienced myself that DFS replication can be extremely slow on Windows 2008 R2.

Once all files are seeded and DFS is configured, the initial replication can still takes days. Replication times are based on:

the number of files
the size of the data
the performance of the disk subsystems of both source and destination

Note: windows 2012 R2 improves DFS replication dramatically, only more reason to upgrade your file servers to 2012 R2 or higher.

If you seed the files, DFS will not transfer files if they are identical, thus this saves bandwidth and time. DFS checks if files differ based on their hash. So even if you seed all data, the initial replication can take a while.

On our virtualised platform, the initial replication of 2.5 GB of data consisting of about five million files took about a full week. To me, that is not a very desirable outcome, but once the initial replication is done, there is no performance issue and all changes are nearly instantly replicated to the secondary server.

For the particular configuration I've setup, the performance storage subsystem could contribute to the slow initial replication.

To speed up the replication process, it's important that you install the latest version of robocopy for Windows 2008 R2 on both systems. There is a bug in older versions of robocopy that do not properly set permissions on files. This results in file hash mismatches, causing DFS to replicate all files, nullifying the benefit of seeding.

Hotfixes for Windows 2008 R2: Hotfixes for Windows 2012 R2:

To verify if a file on both servers has identical hashes, follow these instructions

If you've checked a few files and assured that the hashes are identical, it's ok to configure DFS replication. If you see a lot of Event ID 4412 messages in the DFS Replication event log, there probably is an issue with the file hashes.

How traffic shaping can dramatically improve internet responsiveness

2014-03-08T12:00:00+01:00

At work, access to the internet is provided by a 10 Mbit down / 1 Mbit up ADSL-connection. As we are a mid-size company, bandwidth is clearly a severe constraint. But it was not our biggest problem. Even simple web-browsing was very slow.

As I was setting up a monitoring environment based on Nagios and pnp4nagios, I started to graph the latency of our internet connection, just to prove that we have a problem.

Boy did we get that proof:

Just look at the y-axis, which scale is in milliseconds. For most of the day, the average latency is 175 ms, with some high spikes. Just browsing the web was a pain during times of high-latency, which was clearly almost all of the time.

I became so fed up with our slow internet access that I decided to take matters in my own hands and resolve the low latency issue. The solution? Traffic shaping.

I learned that as ADSL-connections are saturated, especially their upload capacity, you will experience high latency and packet loss. So the trick is to never saturate the connection.

I grabbed a Linux box with two network interfaces and placed it between our internet router and our firewall in bridge mode.

For actual traffic shaping I used wondershaper which is part of Debian or Ubuntu.

apt-get install wondershaper

The wondershaper script is extremely simple, it's specifically build to resolve the problem we face with our ADSL connection. It not only prioritises traffic, it allows you to limit bandwidth usage and thus prevent you from saturating the connection.

This simple example limits bandwidth a bit below full capacity, which dramatically improved latency.

Syntax:

wondershaper <interface> <rx> <tx>

Example:

wondershaper eth1 9500 700

As you can see, latency improved dramatically:

Again, look at the y-axis. We went from an average latency of 175 ms to an average of 35 ms. That's quite an improvement.

Can you spot on which day I implemented traffic shaping?

At the time of writing this blog post, the company is working on fiber internet access, resolving our internet woes, but it will take quite some time before that will be installed, so this is a nice intermediate solution.

Monitoring HP MSA P2000 G3 I/O latency with Nagios

2014-02-04T12:00:00+01:00

At work we have a couple of HP MSA P2000 G3 SANs. These are entry-level SANs that still seem to have almost all features you might want from a SAN, except for official SSD-support.

It seems that the new MSA 2040 adds support for SSDs and also provides 4 GB cache per controller instead of the somewhat meager 2GB of the P2000.

Anyway, a very nice feature of the MSA P2000 G3 is the fact that the management interface also provides a well-documented API that allows you to collect detailed stats on subjects like:

Overall enclosure status
Reports on failed drives
Controller CPU usage
IOPs per controller
IOPs per vdisk
IOPs per disk

Thomas Weaver has written a Nagios plugin that does that: it collects this information and in turn you can graph it with pnp4nagios.

In more recent firmware updates HP has added support to monitor read and write I/O latency per vdisk. Latency is an important indicator for application-level performance so I was quite happy with that.

As the plugin by Thomas did not support reading these parameters yet, I spend some time implementing this check and submitted this new version of check_p2000_api.php back to Thomas.

Read Latency of a RAID 6 array

Write Latency of a RAID 6 array

You will notice that the write latency of this disk array is very high at times, which seem to indicate that this vdisk is taxed too much with too many I/O-requests.

I'd like to thank Thomas Weaver for writing this plugin, I think it's very useful.

Creating a basic ZFS file system on Linux

2014-02-01T12:00:00+01:00

Here are some notes on creating a basic ZFS file system on Linux, using ZFS on Linux.

I'm documenting the scenario where I just want to create a file system that can tollerate at least a single drive failure and can be shared over NFS.

Identify the drives you want to use for the ZFS pool

The ZFS on Linux project advices not to use plain /dev/sdx (/dev/sda, etc.) devices but to use /dev/disk/by-id/ or /dev/disk/by-path device names.

Device names for storage devices are not fixed, so /dev/sdx devices may not always point to the same disk device. I've been bitten by this when first experimenting with ZFS, because I did not follow this advice and then could not access my zpool after a reboot because I removed a drive from the system.

So you should pick the appropriate device from the /dev/disk/by-[id|path] folder. However, it's often difficult to determine which device in those folders corresponds to an actual disk drive.

So I wrote a simple tool called showdisks which helps you identify which identifiers you need to use to create your ZFS pool.

You can install showdisks yourself by cloning the project:

git clone https://github.com/louwrentius/showtools.git

And then just use showdisks like

./showdisks -sp  (-s (size) and -p (by-path) )

For this example, I'd like to use all the 500 GB disk drives for a six-drive RAIDZ1 vdev. Based on the information from showdisks, this is the command to create the vdev:

zfs create tank raidz1 pci-0000:03:00.0-scsi-0:0:21:0 pci-0000:03:00.0-scsi-0:0:19:0 pci-0000:02:00.0-scsi-0:0:9:0 pci-0000:02:00.0-scsi-0:0:11:0 pci-0000:03:00.0-scsi-0:0:22:0 pci-0000:03:00.0-scsi-0:0:18:0

The 'tank' name can be anything you want, it's just a name for the pool.

Please note that with newer bigger disk drives, you should test if the ashift=12 option gives you better performance.

zfs create -o ashift=12 tank raidz1 <devices>

I used this option on 2TB disk drives and the performance and the read performance improved twofold.

How to setup a RAID10 style pool

This is how to create the ZFS equivalent of a RAID10 setup:

zfs create tank mirror <device 1> <device 2> mirror <device 3> <device 4> mirror <device 5> <device 6>

How many drives should I use in a vdev

I've learned to use a 'power of two' (2,4,8,16) of drives for a vdev, plus the appropriate number of drives for the parity. RAIDZ1 = 1 disk, RAIDZ2 = 2 disks, etc.

So the optimal number of drives for RAIDZ1 would be 3,5,9,17. RAIDZ2 would be 4,6,10,18 and so on. Clearly in the example above with six drives in a RAIDZ1 configuration, I'm violating this rule of thumb.

How to disable the ZIL or disable sync writes

You can expect bad throughput performance if you want to use the ZIL / honour synchronous writes. For safety reasons, ZFS does honour sync writes by default, it's an important feature of ZFS to guarantee data integrity. For storage of virtual machines or databases, you should not turn of the ZIL, but use an SSD for the SLOG to get performance to acceptable levels.

For a simple (home) NAS box, the ZIL is not so important and can quite safely be disabled, as long as you have your servers on a UPS and have it cleanly shutdown when the UPS battery runs out.

This is how you turn of the ZIL / support for synchronous writes:

zfs set sync=disabled <pool name>

Disabling sync writes is especially important if you use NFS which issues sync writes by default.

Example:

zfs set sync=disabled tank

How to add an L2ARC cache device

Use showdisks to lookup the actual /dev/disk/by-path identifier and add it like this:

zpool add tank cache <device>

Example:

zpool add tank cache pci-0000:00:1f.2-scsi-2:0:0:0

This is the result (on another zpool called 'server'):

root@server:~# zpool status
  pool: server
 state: ONLINE
  scan: none requested
config:

    NAME                               STATE     READ WRITE CKSUM
    server                             ONLINE       0     0     0
      raidz1-0                         ONLINE       0     0     0
        pci-0000:03:04.0-scsi-0:0:0:0  ONLINE       0     0     0
        pci-0000:03:04.0-scsi-0:0:1:0  ONLINE       0     0     0
        pci-0000:03:04.0-scsi-0:0:2:0  ONLINE       0     0     0
        pci-0000:03:04.0-scsi-0:0:3:0  ONLINE       0     0     0
        pci-0000:03:04.0-scsi-0:0:4:0  ONLINE       0     0     0
        pci-0000:03:04.0-scsi-0:0:5:0  ONLINE       0     0     0
    cache
      pci-0000:00:1f.2-scsi-2:0:0:0    ONLINE       0     0     0

How to monitor performance / I/O statistics

One time sample:

zpool iostat

A sample every 2 seconds:

zpool iostat 2

More detailed information every 5 seconds:

zpool iostat -v 5

Example output:

                                      capacity     operations    bandwidth
pool                               alloc   free   read  write   read  write
---------------------------------  -----  -----  -----  -----  -----  -----
server                             3.54T  7.33T      4    577   470K  68.1M
  raidz1                           3.54T  7.33T      4    577   470K  68.1M
    pci-0000:03:04.0-scsi-0:0:0:0      -      -      1    143  92.7K  14.2M
    pci-0000:03:04.0-scsi-0:0:1:0      -      -      1    142  91.1K  14.2M
    pci-0000:03:04.0-scsi-0:0:2:0      -      -      1    143  92.8K  14.2M
    pci-0000:03:04.0-scsi-0:0:3:0      -      -      1    142  91.0K  14.2M
    pci-0000:03:04.0-scsi-0:0:4:0      -      -      1    143  92.5K  14.2M
    pci-0000:03:04.0-scsi-0:0:5:0      -      -      1    142  90.8K  14.2M
cache                                  -      -      -      -      -      -
  pci-0000:00:1f.2-scsi-2:0:0:0    55.9G     8M      0     70    349  8.69M
---------------------------------  -----  -----  -----  -----  -----  -----

How to start / stop a scrub

Start:

zfs scrub <pool>

Stop:

zfs scrub -s <pool>

Mount ZFS file systems on boot

Edit /etc/defaults/zfs and set this parameter:

ZFS_MOUNT='yes'

How to enable sharing a file system over NFS:

zfs set sharenfs=on <poolname>

How to create a zvol for usage with iSCSI

zfs create -V 500G <poolname>/volume-name

How to force ZFS to import the pool using disk/by-path

Edit /etc/default/zfs and add

ZPOOL_IMPORT_PATH=/dev/disk/by-path/

Links to important ZFS information sources:

Tons of information on using ZFS on Linux by Aaron Toponce:

https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/

Understanding the ZIL (ZFS Intent Log)

http://nex7.blogspot.nl/2013/04/zfs-intent-log.html

Information about 4K sector alignment problems

http://www.opendevs.org/ritk/zfs-4k-aligned-space-overhead.html

Important read about using the proper number of drives in a vdev

http://forums.freenas.org/threads/getting-the-most-out-of-zfs-pools.16/

Why you should not use IPsec for VPN connectivity

2014-01-28T12:00:00+01:00

IPsec is a well-known and widely-used VPN solution. It seems that it's not widely known that Niels Ferguson and Bruce Schneier performed a detailed security analysis of IPsec and that the results were not very positive.

We strongly discourage the use of IPsec in its current form for protection of any kind of valuable information, and hope that future iterations of the design will be improved.

I conveniently left out the second part:

However, we even more strongly discourage any current alterantives, and recommend IPsec when the alternative is an insecure network. Such are the realities of the world.

To put this in context: keep in mind that this paper was released in 2003 and the actual research may even be older (1999!). OpenVPN, an open-source SSL-based VPN solution was born in 2001 and was still maturing in 2003. So there actually was no real alternative back then.

It worries me that this research done by Ferguson and Schneier is more than a decade old. I've been looking for more recent articles on the current security status of IPsec, but I couldn't find much. There have been some new RFCs been published about IPsec but I'm not familiar enough with the material to understand the implications. They make a lot of recommendations in the paper to improve IPsec security, but are they actually implemented?

I did find a presentation from 2013 by Peter Gutmann (University of Auckland). Based on his Wikipedia page, he seems to 'have some knowledge' about cryptography. The paper adresses the Snowden leaks about the NSA and also touches on IPsec. He basically relies on the paper written by Ferguson and Schneier.

But let's think about this: Ferguson and Schneier criticises the design of IPsec. It is flawed by design. That's one of the worst criticisms any thing related to cryptography can get. That design has probably not changed much, from what I understand. So if their critique on IPsec is still mostly valid, all the more reason not to use IPsec.

So this is part of the conclusion and it doesn't beat around the bush:

We have found serious security weaknesses in all major components of IPsec.
As always in security, there is no prize for getting 90% right; you have to get
everything right. IPsec falls well short of that target, and will require some major
changes before it can possibly provide a good level of security.
What worries us more than the weaknesses we have identified is the complexity
of the system. In our opinion, current evaluation methods cannot handle
systems of such a high complexity, and current implementation methods are not
capable of creating a secure implementation of a system as complex as this.

So if not IPsec, what should you use? I would opt to use an SSL/TLS-based VPN solution like OpenVPN.

I can't vouch for the security for OpenVPN, but a well-known Dutch security firm Fox-IT has released a stripped-down version of the OpenVPN software (removed features) that they consider fit for (Dutch) governmental use. Not to say that you should use that particular OpenVPN version: the point is that OpenVPN is deemed secure enough to be used for governmental usage. For whatever that's worth.

At least, SSL-based VPN solutions have the benefit that they use SSL/TLS, which may have it's own problems, but is at least not as complex as IPsec.

Achieving 450 MB/s network file transfers using Linux Bonding

2014-01-07T01:00:00+01:00

Linux Bonding

In this article I'd like to show the results of using regular 1 Gigabit network connections to achieve 450 MB/s file transfers over NFS.

I'm again using Linux interface bonding for this purpose.

Linux interface bonding can be used to create a virtual network interface with the aggregate bandwidth of all the interfaces added to the bond. Two gigabit network interfaces will give you - guess what - two gigabit or ~220 MB/s of bandwidth.

This bandwidth can be used by a single TCP-connection.

So how is this achieved? The Linux bonding kernel module has special bonding mode: mode 0 or round-robin bonding. In this mode, the kernel will stripe packets across the interfaces in the 'bond' like RAID 0 with hard drives. As with RAID 0, you get additional performance with each device you add.

So I've added HP NC364T quad-port network cards to my servers. Thus each server has a theoretical bandwidth of 4 Gigabit. These HP network cards cost just 145 Euro and I even found the card for 105 Dollar on Amazon.

With two servers, you just need four UTP cable's to connect the two network interfaces and you're done. This would cost you ~300 Euro or ~200 dollar in total.

If you want to connect additional servers, you need a managed gigabit switch with VLAN-support and sufficient ports. Each additional server will use 4 ports on the switch, excluding interfaces for remote access and other purposes.

Managed gigabit switches are quite inexpensive these days. I bought a 24 port switch: HP 1810-24G v2 (J9803A) for about 180 euros (209 Dollars on Newegg) and it can even be rack-mounted.

Using VLANS

So you can't just use a gigabit switch and connect all these quad-port network cards to a single VLAN. I tested this scenario first and only got a maximum transfer speed of 270 MB/s while copying a file between servers over NFS.

The trick is to create a separate VLAN for every network port. So if you use a quad-port network card, you need four VLANs. Also, you must make sure that every port on every network card is in the same VLAN. For example, port 1 on every card needs to be in VLAN 21, port 2 in VLAN 22, and so on. You also must add the appropriate switch port to the correct VLAN. Last, you must add the network interfaces to the bond in the right order.

So why do you need to use VLANs? The reason is quite simple. Bonding works by spoofing the same hardware or MAC-address on all interfaces. So the switch sees the same hardware address on four ports, and thus gets confused. To which port should the packet be sent?

If you put each port in it's own VLAN, the 'spoofed' MAC-address is seen only once in each VLAN. So the switch won't be confused. What you are in fact doing by creating VLANs is creating four separate switches. So if you have - for example - four cheap 8-port unmanaged gigabit switches, this would work too.

So assuming that you have four ethernet interfaces, this is an example of how you can create the bond:

ifenslave bond0 eth1 eth2 eth3 eth4

Next, you just assign an IP-address to the bond0 interface, like you would with a regular eth(x) interface.

ifconfig bond0 192.168.2.10 netmask 255.255.255.0

Up to this point, I only achieved about 350 MB/s. I needed to enable jumbo frames on all interfaces and the switch to achieve 450 MB/s.

ifconfig bond0 mtu 9000 up

Next, you can just mount any NFS share over the interface and start copying files. That's all.

Once I added each interface to the appropriate VLAN, I got about 450 MB/s for a single file copy with 'cp' over NFS.

I did not perform a 'cp' but a 'dd' because I don't have a disk array fast enough (yet) that can write at 450 MB/s.

So for three servers, this solution will cost me 600 Euro or 520 Dollar.

What about LAGs and LACP?

Don't configure your clients or switches for LACP, it doesn't give you the speed benefit and it's not required.

10 Gigabit ethernet?

Frankly, it's quite expensive if you need to connect more than two servers. An entry-level 10Gbe NIC like the Intel X540-T1 does about 300 Euros or 450 Dollar. This card allows you to use Cat 6e UTP Cabling. (Pricing from Dutch Webshops in Euros and Newegg in Dollars).

With two of those, you can have 10Gbit ethernet between two servers for 600 Euro or 700 Dollar. If you need to connect more servers, you would need a switch. The problem is that 10Gbit switches are not cheap. An 8-port unmanaged switch from Netgear (ProSAFE Plus XS708E) does about 720 Euro's or 900 Dollar.

If you want to connect three servers, you need three network cards and a switch. So three network cards and a switch will cost you 900 Euro (1050 Dollar) for the network cards and 720 Euro (900 Dollar) for the switch, totalling 1800 euro or 1950 Dollar.

You will get higher transfer speeds, but at a significantly higher price.

For many business purposes this higher price can be easily justified and I would select 10 Gb over 1 Gb bonding in a heart-beat. Less cables, higher performance, lower latency.

However, bonding gigabit interfaces allows you to use off-the-shelf equipment and maybe a nice compromis between cost, usability and performance.

Operating System Support

As far as I'm aware, round-robin bonding is only supported on Linux. Other operating systems do not support it.

Things you should consider when building a ZFS NAS

2013-12-29T12:00:00+01:00

ZFS is a modern file system designed by Sun Microsystems, targeted at enterprise environments. Many features of ZFS also appeal to home NAS builders and with good reason. But not all features are relevant or necessary for home use.

I believe that most home users building their own NAS, are just looking for a way to create a large centralised storage pool. As long as the solution can saturate gigabit ethernet, performance is not much of an issue. Workloads are typically single-client and sequential in nature.

If this description rings true for your environment, there are some options regarding ZFS that are often very popular but not very relevant to you. Furthermore, there are also some facts that you should take into account when preparing for your own NAS build.

Expanding your pool may not be as simple as you think

Added August 2015

If you are familiar with regular hardware or software RAID, you might expect to use on-line capacity expansion, where you just grow a RAID array with extra drives as you see fit. Many hardware RAID cards support this feature and Linux software RAID (MDADM) supports this too. This is very economical: just add an extra drive and you gain some space. Very flexible and simple.

But ZFS does not support this approach. ZFS requires you to create a new 'RAID array' and add it to the pool. So you will lose extra drives to redundancy.

To be more presise: you cannot expand VDEVs. You can only add VDEVS to a pool. And each VDEV requires it's own redundancy.

So if you start with a single 6-disk RAIDZ2 you may end up with two 6-disk RAIDZ2 VDEVS. This means you use 4 out of 12 drives for redundancy. If you would have started out with a 10-disk RAIDZ2, you would only lose 2 drives to redundancy. Example:

A: 2 x 6-disk RAIDZ2 consisting of 4 TB drives = 12 disks - 4 redundancy = 8 x 4 = 32 TB netto capacity.

B: 1 x 10-disk RAIDZ2 consisting of 4 TB drives = 10 disks - 2 redundancy = 8 x 2 = 32 TB netto capacity.

Option A lost you two drives at 150$ = 300$ and also requires 2 extra SATA ports and chassis slots.

Option B will cost you 4 extra drives upfront, space you may not need immediately.

Also take into account that there is such a thing as a 'recommended number of disks in a VDEV' depending on the redundancy used. This is discussed further down.

Tip: there is a 'trick' how you can expand a VDEV. If you replace every drive with a larger one, you can then resize the VDEV and the pool. So replacing 2 TB drives with 4 TB drives would double your capacity without adding an extra VDEV.

This approach requires a full VDEV rebuild after each drive replacement. So you may understand that this takes quite some time, during which your are running on no (RAIDZ) or less (RAIDZ2) redundancy. But it does work.

If you have an additional spare SAS/SATA port and power, you can keep the redundancy and do an 'on-line' replace of the drive. This way, you don't lose or reduce redundancy during a rebuild. This is relatively ideal if you also have room for an additional drive in the chassis.

This can be quite some work if you do have an available SATA port, but no additional drive slots. You will have to open the chassis, find a temporary spot for the new drive and then after the rebuild, move the new drive into the slot of the old one.

There is a general recommendation not to mix different VDEV sizes in a pool, but for home usage, this is not an issue. So you could - for example - expand a pool based on a 6-drive VDEV with an additional 4-drive VDEV RAIDZ2.

Remember: lose one VDEV and you lose your entire pool. So I would not recommend mixing RAIDZ and RAIDZ2 VDEVs in a pool.

You don't need a SLOG for the ZIL

Quick recap: the ZIL or ZFS intent Log is - as I understand it - only relevant for synchronous writes. If data integrity is important to an application, like a database server or a virtual machine, writes are performed synchronous. The application wants to make sure that the data is actually stored on the physical storage media and it waits for a confirmation from ZFS that it has done so. Only then will it continue.

Asynchronous writes on the contrary, never hit the ZIL. They are just cached in RAM and directly written to the VDEV in one sequential swoop when the next transaction group commit will be performed (currently by default every 5 seconds). In the mean time, the application gets a confirmation of ZFS that the data is stored (a white lie) and just continues where it left off. ZFS just caches the write in memory and actually write the data to the storage VDEV when it feels like it (fifo).

As you may understand, asynchronous writes are way faster because they can be cached and ZFS can reorder the I/O to make it more sequential and prevent random I/O from hitting the VDEV. This is what I understood from this source.

So if you encounter synchronous writes, they must be committed to the ZIL (thus VDEV) and this causes random I/O patterns on the VDEV, degrading performance significantly.

The cool thing about ZFS is that it does provide the option to store the ZIL on a dedicated device called the SLOG. This doesn't do anything for performance by itself, but the secret ingredient is using a solid state drive as the SLOG, ideally in a mirror to insure data integrity and to maintain performance in the case of a SLOG device failure.

For business critical environments, a separate SLOG device based on SSDs is a no-brainer. But for home use? If you don't have a SLOG, you still have a ZIL, it's only not as fast. That's not a real problem for single-client sequential throughput.

For home usage, you may even consider how much you care about data integrity. That sounds strange, but the ZIL is used to recover from the event of a sudden power-loss. If your NAS is attached to a UPS, this is not much of a risk, you can perform a controlled shutdown before the batteries run out of power. The remaining risk is human error or some other catastrophic event within your NAS.

So all data in rest already stored on your NAS is never at risk. It's only data that is in the process of being committed to storage that may get scrambled. But again: this is a home situation. Maybe restart your file transfer and you are done. You still have a copy of the data on the source device. This is entirely different from a setup with databases or virtual machines.

Data integrity of data at rest is vitally important. The ZIL only protects data in transit. It has nothing to do with the data already committed to the VDEV.

I see so many NAS builders being talked into buying some specific SSDs to be used for the ZIL whereas they probably won't benefit from them at all, it's just too bad.

You don't need L2ARC cache

ZFS relies heavily on caching of data to deliver decent performance, especially read performance. RAM provides the fasted cache and that is where the first level of caching lives, the ARC (Adaptive Replacement Cache). ZFS is smart and learns which data is often requested and keeps it in the ARC.

But the size of the ARC is limited by the amount of RAM available. This is why you can add a second cache tier, based on SSDs. SSDs are not as fast as RAM, but still way faster than spinning disks. And they are cheaper compared to RAM memory if you look at their capacity.

For additional more detailed information, go to this site

L2ARC is important when you have multiple users or VMs accessing the same data sets. In this case, L2ARC based on SSDs will improve performance significantly. But if we just take a look at the average home NAS build, I'm not sure how the L2ARC adds any benefit. ZFS has no problem with single-client sequential file transfers so there is no benefit in implementing a L2ARC.

Update 2015-02-08: There is even a downside to having a L2ARC cache. All the meta-data regarding data stored in the L2ARC cache is kept in memory, and thus eating away at your ARC!, thus your ARC becomes less effective (source).

You don't need deduplication and compression

For the home NAS, most data you store on it is already highly compressed and additional compression only wastes performance (Music, Videos, etc). It is a cool feature, but not so much for home use. If you are planning to store other types of data, compression actually may be of interest (documents, backups of VMs, etc). It is suggested by many (and in the comments) that with LZ4 compression, you don't lose performance (except for some CPU cycles) and with compressible data, you even gain performance, so you could just enable it and forget about it.

Whereas compression may do not much harm, Deduplication is often more relevant in business environments where users are sloppy and store multiple copies of the same data at different locations. I'm quite sure you don't want to sacrifice RAM and performance for ZFS to keep track of duplicates you probably don't have.

You don't need an ocean of RAM

The absolute minimum RAM for a viable ZFS setup is 4 GB but there is not a lot of headroom for ZFS here. ZFS is quite memory hungry because it uses RAM as a buffer so it can perform operations like checksums and reorder all I/O to be sequential.

If you don't have sufficient buffer memory, performance will suffer. 8 GB is probably sufficient for most arrays. If your array is faster, more memory may be required to actually benefit from this performance. For maximum performance, you should have enough memory to hold 5 seconds worth of maximum write throughput ( 5 x 400MB/s = 2GB ) and leave sufficient headroom for other ZFS RAM requirements. In the example, 4 GB RAM could be sufficient.

For most home users, saturating gigabit is already sufficient so you might be safe with 8 GB of RAM in most cases. More RAM may not provide much more benefit, but it will increase power consumption.

There is an often cited rule that you need 1 GB of RAM for every TB of storage, but this is not true for home NAS solutions. This is only relevant for high-performance multi-user or multi-VM environments.

Additional information about RAM requirements can be found here

You do need ECC RAM if you care about data integrity

The money saved on a ZIL or L2ARC cache can be better spend on ECC RAM memory.

ZFS does not rely on the quality of individual disks. It uses parity to verify that disks don't lie about the data stored on them (data corruption).

But ZFS can't verify the contents of RAM memory, so here ZFS relies on the reliability of the hardware. And there is a reason why we use RAID or redundant power suplies in our server equipment: hardware fails. RAM fails too. This is the reason why every server product by well-known vendors like HP, Dell, IBM and Supermicro only support ECC memory. RAM memory errors do occur more frequent than you may think.

ECC (Error Checking and Correcting) RAM corrects and detects RAM errors. This is the only way you can be fairly sure that ZFS is not fed with corrupted data. Keep in mind: with bad RAM, it is likely that corrupted data will be written to disk without ZFS ever being aware of it (garbage in - garbage out).

Please note that the quality of your RAM memory will not directly affect any data that is at rest and already stored on your disks. Existing data will only be corrupted with bad RAM if it is modified or moved around. ZFS will probably detect checksum errors, but it will be too late by then...

To me, it's simple. If you care enough about your data that you want to use ZFS, you should also be willing to pay for ECC memory. You are giving yourself a false sense of security if you do not use ECC memory. ZFS was never designed for consumer hardware, it was destined to be used on server hardware using ECC memory. Because it was designed with data integrity as the top most priority.

There are entry-level servers that do support ECC memory and can be had fairly cheap with 4 hard drive bays, like the HP ProLiant MicroServer Gen8.

I wrote an article about a reasonably priced CPU+RAM+MB combo that does support ECC memory starting at $360.

If you feel lucky, go for good-quality non-ECC memory. But do understand that you are taking a risk here.

Understanding random I/O performance

Added August 2015

With ZFS, the rule of thumb is this: regardless of the number of drives in a RAIDZ(2/3) VDEV, you always get roughly the random I/O performance of a single drive in the VDEV¹.

Now I want to make the case here that if you are building your own home NAS, you shouldn't care about random I/O performance too much.

If you want better random I/O performance of your pool, the way to get it is to:

add more VDEVS to your pool
add more RAM/L2ARC for caching
use disks with higher RPM or SSDs combined with option 1.

Regarding point 1:

So if you want the best random I/O performance, you should just use a ton of mirrored drives in the VDEV, so you essentially create a large RAID 10. This is not very space-efficient, so probably not so relevant in the context of a home NAS.

Example similar to RAID 10:

root@bunny:~# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool  59.5K  8.92T    19K  /testpool

root@bunny:~# zpool status
  pool: testpool
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    testpool    ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
      mirror-2  ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0
      mirror-3  ONLINE       0     0     0
        sdi     ONLINE       0     0     0
        sdj     ONLINE       0     0     0
      mirror-4  ONLINE       0     0     0
        sdk     ONLINE       0     0     0
        sdl     ONLINE       0     0     0
      mirror-5  ONLINE       0     0     0
        sdm     ONLINE       0     0     0
        sdn     ONLINE       0     0     0
      mirror-6  ONLINE       0     0     0
        sdo     ONLINE       0     0     0
        sdp     ONLINE       0     0     0
      mirror-7  ONLINE       0     0     0
        sdq     ONLINE       0     0     0
        sdr     ONLINE       0     0     0
      mirror-8  ONLINE       0     0     0
        sds     ONLINE       0     0     0
        sdt     ONLINE       0     0     0
      mirror-9  ONLINE       0     0     0
        sdu     ONLINE       0     0     0
        sdv     ONLINE       0     0     0

Another option, if you need better storage efficiency, is to use multiple RAIDZ or RAIDZ2 VDEVS in the pool. In a way, you're then creating the equivalent of a RAID50 or RAID60.

Example similar to RAID 50:

root@bunny:~# zfs list
NAME       USED  AVAIL  REFER  MOUNTPOINT
testpool  77.5K  14.3T  27.2K  /testpool

root@bunny:~# zpool status
  pool: testpool
 state: ONLINE
  scan: none requested
config:

NAME        STATE     READ WRITE CKSUM
testpool    ONLINE       0     0     0
  raidz1-0  ONLINE       0     0     0
    sdc     ONLINE       0     0     0
    sdd     ONLINE       0     0     0
    sde     ONLINE       0     0     0
    sdf     ONLINE       0     0     0
    sdg     ONLINE       0     0     0
  raidz1-1  ONLINE       0     0     0
    sdh     ONLINE       0     0     0
    sdi     ONLINE       0     0     0
    sdj     ONLINE       0     0     0
    sdk     ONLINE       0     0     0
    sdl     ONLINE       0     0     0
  raidz1-2  ONLINE       0     0     0
    sdm     ONLINE       0     0     0
    sdn     ONLINE       0     0     0
    sdo     ONLINE       0     0     0
    sdp     ONLINE       0     0     0
    sdq     ONLINE       0     0     0
  raidz1-3  ONLINE       0     0     0
    sdr     ONLINE       0     0     0
    sds     ONLINE       0     0     0
    sdt     ONLINE       0     0     0
    sdu     ONLINE       0     0     0
    sdv     ONLINE       0     0     0

You only need to deploy these kinds of pool/vdev configuratoins if you have valid reason that you need the random I/O performance they provide. Creating less but larger VDEVs is often more space efficient and will still saturate gigabit when transferring large files.

It's ok to use multiple VDEVs of different drive sizes

This only true in the context of a home NAS.

Let's take an example. You have an existing pool consisting of a single RAIDZ VDEV with 4 x 2 TB drives and your pool is filling up.

It's then perfectly fine in the context of a home NAS to add a second VDEV consisting of a 5 x 4 TB RAIDZ.

ZFS will take care of how data is distributed across the VDEVs.

It is NOT recommended to mix different RAIDZ schemas, so VDEV 1 = RAIDZ and VDEV 2 = RAIDZ2. Remember that losing a single VDEV = losing the whole pool. It doesn't make sense to mix redundancy levels.

VDEVs should consist of the optimal number of drives

Added August 2015: If you use the large_blocks feature and use 1MB records, you don't need to adhere to the rule of always putting a certain number of drives in a VDEV to prevent significant loss of storage capacity.

This enables you to create an 8-drive RAIDZ2 where normally you would have to create either a RAIDZ2 VDEV that consists of 6 drives or 10 drives.

For home use, expanding storage by adding VDEVs is often suboptimal because you may spend more disks on redundancy than required, as explained earlier. The support of large_blocks allows you to buy the number of disks upfront that suits current and future needs.

In my own personal case, with my 19" chassis filled with 24 drives, I would enable the large_blocks feature and create a single 24-drive RAID-Z3 VDEV to give me optimal space and still very good redundancy.

The large_blocks feature is supported on ZFS on Linux since version 0.6.5 (September 2015).

Thanks to user "SirMaster" on Reddit for introducing this feature to me.

Original advice:

Depending on the type of 'RAID' you may choose for the VDEV(s) in your ZFS pool, you might want to make sure you only put in the right number of disks in the VDEV.

This is important, if you don't use the right amount, performance will suffer, but more importantly: you will lose storage space, which can ad up to over 10% of the available capacity. That's quite a waste.

This is a straight copy&paste from sub.mesa's post

The following ZFS pool configurations are optimal for modern 4K sector harddrives:
RAID-Z: 3, 5, 9, 17, 33 drives
RAID-Z2: 4, 6, 10, 18, 34 drives
RAID-Z3: 5, 7, 11, 19, 35 drives

Sub.mesa also explains the details on why this is true. And here is another example.

The gist is that you must use a power of two for your data disks and then add the number of parity disks required for your RAIDZ level on top of that. So 4 data disks + 1 parity disk (RAIDZ) is a total of 5 disks. Or 16 data disks + 2 parity disks (RAIDZ2) is 18 disks in the VDEV.

Take this into account when deciding on your pool configuration. Also, RAIDZ2 is absolutely recommended. with more than 6-8 disks. The risk of losing a second drive during 'rebuild' (resilvering) is just too high with current high-density drives.

You don't need to limit the number of data disks in a VDEV

For home use, creating larger VDEVs is not an issue, even an 18 disk VDEV is probably fine, but don't expect any significant random I/O performance. It is always recommended to use multiple smaller VDEVs to increase random I/O performance (at the cost of capacity lost to parity) as ZFS does stripe I/O-requests across VDEVs. If you are building a home NAS, random I/O is probably not very relevant.

You don't need to run ZFS at home

ZFS is cool technology and it's perfectly fine to run ZFS at home. However, the world doesn't end if you don't.

https://blogs.oracle.com/roch/entry/when_to_and_not_to ↩

ZFS on Linux: monitor cache hit ratio

2013-12-23T12:00:00+01:00

I'm performing some FIO random read 4k I/O benchmarks on a ZFS file system. So since I didn't trust the numbers I got, I wanted to know how many of the IOPs I got were due to cache hits rather than disk hits.

This is why I wrote a small shell script called archhitratio.

Sample output:

IOPs: 133 | ARC cache hit ratio: 48.00 % | Hitrate: 64 / Missrate: 69
IOPs: 131 | ARC cache hit ratio: 48.00 % | Hitrate: 63 / Missrate: 68
IOPs: 136 | ARC cache hit ratio: 49.00 % | Hitrate: 67 / Missrate: 69
IOPs: 128 | ARC cache hit ratio: 46.00 % | Hitrate: 59 / Missrate: 69
IOPs: 127 | ARC cache hit ratio: 46.00 % | Hitrate: 59 / Missrate: 68
IOPs: 135 | ARC cache hit ratio: 48.00 % | Hitrate: 65 / Missrate: 70
IOPs: 127 | ARC cache hit ratio: 45.00 % | Hitrate: 58 / Missrate: 69
IOPs: 125 | ARC cache hit ratio: 44.00 % | Hitrate: 56 / Missrate: 69
IOPs: 128 | ARC cache hit ratio: 46.00 % | Hitrate: 60 / Missrate: 68

In this example, I'm performing a random read test on a 16GB data set. This host has 16 GB RAM and 6 GB of this dataset was already in memory from previous FIO runs. This is why we see a ~45% hit ratio.

This is a more interesting result:

IOPs: 1404 | ARC cache hit ratio: 90.0 % | Hitrate: 1331 / Missrate: 73
IOPs: 1425 | ARC cache hit ratio: 90.0 % | Hitrate: 1350 / Missrate: 75
IOPs: 1395 | ARC cache hit ratio: 90.0 % | Hitrate: 1323 / Missrate: 72
IOPs: 1740 | ARC cache hit ratio: 90.0 % | Hitrate: 1664 / Missrate: 76
IOPs: 1351 | ARC cache hit ratio: 90.0 % | Hitrate: 1277 / Missrate: 74
IOPs: 1613 | ARC cache hit ratio: 90.0 % | Hitrate: 1536 / Missrate: 77
IOPs: 1920 | ARC cache hit ratio: 90.0 % | Hitrate: 1845 / Missrate: 75
IOPs: 1431 | ARC cache hit ratio: 90.0 % | Hitrate: 1354 / Missrate: 77
IOPs: 1675 | ARC cache hit ratio: 90.0 % | Hitrate: 1598 / Missrate: 77
IOPs: 1560 | ARC cache hit ratio: 90.0 % | Hitrate: 1484 / Missrate: 76
IOPs: 1574 | ARC cache hit ratio: 90.0 % | Hitrate: 1500 / Missrate: 74
IOPs: 2017 | ARC cache hit ratio: 90.0 % | Hitrate: 1946 / Missrate: 71
IOPs: 1696 | ARC cache hit ratio: 90.0 % | Hitrate: 1623 / Missrate: 73
IOPs: 1776 | ARC cache hit ratio: 90.0 % | Hitrate: 1702 / Missrate: 74
IOPs: 1671 | ARC cache hit ratio: 90.0 % | Hitrate: 1597 / Missrate: 74
IOPs: 1729 | ARC cache hit ratio: 90.0 % | Hitrate: 1656 / Missrate: 73
IOPs: 1902 | ARC cache hit ratio: 90.0 % | Hitrate: 1828 / Missrate: 74
IOPs: 2029 | ARC cache hit ratio: 90.0 % | Hitrate: 1956 / Missrate: 73
IOPs: 2228 | ARC cache hit ratio: 90.0 % | Hitrate: 2161 / Missrate: 67
IOPs: 2289 | ARC cache hit ratio: 90.0 % | Hitrate: 2216 / Missrate: 73
IOPs: 2385 | ARC cache hit ratio: 90.0 % | Hitrate: 2277 / Missrate: 108
IOPs: 2595 | ARC cache hit ratio: 90.0 % | Hitrate: 2524 / Missrate: 71
IOPs: 2940 | ARC cache hit ratio: 90.0 % | Hitrate: 2872 / Missrate: 68
IOPs: 2984 | ARC cache hit ratio: 90.0 % | Hitrate: 2872 / Missrate: 112
IOPs: 2622 | ARC cache hit ratio: 90.0 % | Hitrate: 2385 / Missrate: 237
IOPs: 1518 | ARC cache hit ratio: 90.0 % | Hitrate: 1461 / Missrate: 57
IOPs: 3221 | ARC cache hit ratio: 90.0 % | Hitrate: 3150 / Missrate: 71
IOPs: 3745 | ARC cache hit ratio: 90.0 % | Hitrate: 3674 / Missrate: 71
IOPs: 3363 | ARC cache hit ratio: 90.0 % | Hitrate: 3292 / Missrate: 71
IOPs: 3931 | ARC cache hit ratio: 90.0 % | Hitrate: 3856 / Missrate: 75
IOPs: 3765 | ARC cache hit ratio: 90.0 % | Hitrate: 3689 / Missrate: 76
IOPs: 4845 | ARC cache hit ratio: 90.0 % | Hitrate: 4772 / Missrate: 73
IOPs: 4422 | ARC cache hit ratio: 90.0 % | Hitrate: 4350 / Missrate: 72
IOPs: 5602 | ARC cache hit ratio: 90.0 % | Hitrate: 5531 / Missrate: 71
IOPs: 5351 | ARC cache hit ratio: 90.0 % | Hitrate: 5279 / Missrate: 72
IOPs: 6075 | ARC cache hit ratio: 90.0 % | Hitrate: 6004 / Missrate: 71
IOPs: 6586 | ARC cache hit ratio: 90.0 % | Hitrate: 6515 / Missrate: 71
IOPs: 7974 | ARC cache hit ratio: 90.0 % | Hitrate: 7907 / Missrate: 67
IOPs: 4434 | ARC cache hit ratio: 90.0 % | Hitrate: 4180 / Missrate: 254
IOPs: 9793 | ARC cache hit ratio: 90.0 % | Hitrate: 9721 / Missrate: 72
IOPs: 9395 | ARC cache hit ratio: 90.0 % | Hitrate: 9300 / Missrate: 95
IOPs: 6171 | ARC cache hit ratio: 90.0 % | Hitrate: 6089 / Missrate: 82
IOPs: 9209 | ARC cache hit ratio: 90.0 % | Hitrate: 9142 / Missrate: 67
IOPs: 14883 | ARC cache hit ratio: 90.0 % | Hitrate: 14817 / Missrate: 66
IOPs: 11304 | ARC cache hit ratio: 90.0 % | Hitrate: 11152 / Missrate: 152
IOPs: 228 | ARC cache hit ratio: 30.0 % | Hitrate: 71 / Missrate: 157
IOPs: 8321 | ARC cache hit ratio: 90.0 % | Hitrate: 8072 / Missrate: 249
IOPs: 15550 | ARC cache hit ratio: 90.0 % | Hitrate: 15450 / Missrate: 100
IOPs: 11819 | ARC cache hit ratio: 90.0 % | Hitrate: 11683 / Missrate: 136
IOPs: 28630 | ARC cache hit ratio: 90.0 % | Hitrate: 28367 / Missrate: 263
IOPs: 40484 | ARC cache hit ratio: 90.0 % | Hitrate: 40409 / Missrate: 75
IOPs: 104501 | ARC cache hit ratio: 90.0 % | Hitrate: 103982 / Missrate: 519
IOPs: 164483 | ARC cache hit ratio: 90.0 % | Hitrate: 163997 / Missrate: 486
IOPs: 229729 | ARC cache hit ratio: 90.0 % | Hitrate: 228956 / Missrate: 773
IOPs: 236479 | ARC cache hit ratio: 90.0 % | Hitrate: 235886 / Missrate: 593
IOPs: 249232 | ARC cache hit ratio: 90.0 % | Hitrate: 248836 / Missrate: 396
IOPs: 259156 | ARC cache hit ratio: 90.0 % | Hitrate: 258968 / Missrate: 188
IOPs: 276099 | ARC cache hit ratio: 90.0 % | Hitrate: 275857 / Missrate: 242
IOPs: 249382 | ARC cache hit ratio: 90.0 % | Hitrate: 249287 / Missrate: 95

What does this result mean? The RAM size is 16 GB and the test data size is only 6 GB. If you just continue performing random I/O, eventually all data will be in RAM. I believe that here, you witness the moment when all data is in RAM and the already high IOPs goes through the roof (250K IOPS). However, I cannot explain the increase of the Missrate.

An affordable server platform based on server-grade hardware

2013-12-13T12:00:00+01:00

Updated post (June 2014) found HERE

There are some reasons why you should consider buying true server-grade harware when building a server, wether it's for home or business use.

This is why I want to introduce you to the Supermicro X9SCM-F motherboard.

This motherboard costs about $160 or €160, which is more than a desktop grade motherboard, but you get quite a lot in return.

Remote KVM over IP

First of all, this motherboard has a separate network interface dedicated to an on-board Keyboard-Video-Mouse interface (IPMI). This interface allows you to power the server on or off, enter the BIOS all through a web interface, through the network. You never need to be in the vicinity of your server unless you need to perform some hardware maintenance.

Support for ECC (Error-correcting Code) RAM

All server grade hardware from manufactures like Supermicro, Dell or HP all ship their servers with ECC RAM. This type of memory is more expensive than regular RAM, but based on the name, you may guess the benefit: it detects and corrects memory errors. If you truly care about data integrity and availability, this is a recommended feature.

Support for both cheap or faster and more expensive processors

Cheap processor with ECC RAM support

The cheapest solution seems to be the Intel Pentium G2030 @ 2.00 GHz. For only 72 dollar or 52 Euro it can be yours and it supports ECC memory. Performance is not stellar but sufficient for most NAS builds. More proof that a build based on ECC does not have to be expensive.

One step up the ladder is the Intel Core i3-3220. It retails for about $124 or €125 and it supports ECC memory, something I didn't suspect of the i3 series. At 3.3Ghz it's quite fast and it is a dual-core processor with hyper-threading support. The processor also supports virtualisation (VT-x), but it does not support I/O virtualisation (VT-d).

Faster processor with full virtualisation support

At $229 or €200 the Intel Xeon E3-1230 V2 cost you twice the price of the Core i3, but you get twice the processor cores (quad-core) with hyper-threading, turbo-boost up to 3.7 ghz and full virtualisation support (both VT-x and VT-d). For the money, you get a beast of a processor with ECC RAM support.

Overview of CPU's

CPU	Passmark score	Price in Euro	Price in Dollars
Intel Pentium G2030 @ 3.00GHz	3008	52 Euro	72 Dollar*
Intel Core i3-3220 @ 3.30GHz	4226	97 Euro	125 Dollar
Intel Xeon E3-1230 V2 @ 3.30GHz	8890	196 Euro	230 Dollar

Dollars are from Newegg (* is estimate).
Euros are including taxes.

Support for 32 GB RAM

32 GB is quite some RAM and the maximum amount supported by the free version of VMware ESXi (ESXi 5.5 does not have this limit I've been told). Basically, it supports entry-level virtualisation needs or web-application needs. The 32 GB RAM limit is one of the biggest constraints of this platform, but may be sufficient for most applications except for large virtualisation loads or big databases.

32 GB is more than enough for everyone who likes to build a beefy storage server.

Expandability: 4 x PCI-e 8x (physical size) slots

This is probably why the Supermicro X9SCM-F motherboard is so interesting for NAS or storage builders. It has four PCI-e slots in a 8x physical form factor. Two of them are PCI-e 2.0 4x in a physical 8x slot and the other two are true PCI-e 3.0 8x slots. But even a PCI-e 2.0 4x slot provides you with 4 x 500 MB/s = 2000 MB/s per slot. The 8x slots have 8 lanes at 985 MB/s each thus totalling almost 8 GB/s! Let's be honest: the PCI-e 2.0 4x provide more than enough bandwidth to power any disk controller or network interface.

Because there are four slots, there are several possibilities. For example, you could populate them with four IBM M1015 HBA's providing you with 32 SAS/SATA ports for a total of 32 disks. With the six on-board SATA ports, you could theoretically connect 38 disks.

If you put only 3 x IBM M1015 HBA's in the motherboard you will leave room for fast networking, like a quad-port gigabit network card. You can then put the four gigabit ports into Linux interface bonding to achieve 400 MB/s network transfer speeds.

You could also consider to implement 10GB network cards (more expensive) or take a look into fibre channel or infiniband network connectivity.

With those four PCI-e slots, you can come a very long way.

Two on-board Gigabit network interfaces

The motherboard supports two on-board network interfaces and you can use them to bond them together for extra performance and/or fail-over.

You could also use one interface for regular LAN traffic and the other one for iSCSI or other protocols.

Conclusion

I believe the Supermicro X9SCM-F motherboard is a very interesting all-round platform for any type of server and particularly interesting as a platform for storage servers.

If you take the motherboard ($160), 8 GB RAM ($75) and entry level processor ($125) you get a server-grade platform for $360. The same platform with 16 GB RAM will cost you about $435. The prices in Euros will be about the same.

Eztables: simple yet powerful firewall configuration for Linux

2013-11-16T12:00:00+01:00

I've created and released Eztables on Github. Anyone who ever has a need to setup a firewall on Linux may be interested in this project.

It doesn't matter if you need to protect a laptop, server or want to setup a network firewall. Eztables supports it all.

If you're not afraid to touch the command line and edit a text file, you may be quite pleased with Eztables.

Go check it out!

Script that shows smart values of all disks'

2013-10-05T01:00:00+02:00

Please use this tool on github instead of this ancient script.

So you have a Linux system with a lot of hard drives. If you want to quickly check on some key SMART values to determine the health of individual disks, you might be interested in this script.

I wrote a small Python script called 'showsmart' that displays key SMART values for all disks that support SMART. This has been tested on Ubuntu and Debian.

REALLOC. = Reallocated Sector Count
PENDING = Current Pending Sector
CRC ERR. = UDMA CRC ERROR

This script only requires python and smartctl (smartmontools).

I hope someone will find it useful.

Script that shows relevant disk information

2013-10-02T01:00:00+02:00

I wrote a small Python script called 'showdisks' that displays relevant information about any physical storage devices supported by hdparm.

Information such as model and capacity are shown, but also controller and device path.

This script only requires python and hdparm.

I hope someone will find it useful.

Finding a good blu-ray player for Mac OS X

2013-09-22T18:00:00+02:00

I find playing a Blu-ray movie on my Mac cumbersome. I've been using Plex, XBMC and VLC but these free open-source products are all a usability nightmare.

To play a Blu-ray movie, you have to perform these steps:

right-click on the BDMV file
choose 'show packet contents'
go to the STREAM folder
sort the files by size (from large to small)
select the biggest m2ts file and open it in the appropriate player

I got so fed-up with this process that I started searching for any product that just allows me to point it to a folder with Blu-ray content and friggin' play it. Fortunately, there is such a product for Mac OS X, it's called Mac Blu-ray Player and it's a paid application. I paid 36 euro (48 dollar) including taxes.

Those 36 euros are well spend. No it is not free, no it is not open-source, but I don't care. It is a good product. If you value your time I highly recommend buying this software.

I don't have anything against free or open-source software, but I do have a grudge against software that is not user-friendly. If software is not easy to use, something you would expect from a media player, it's broken.

Fortunately, you don't have to trust me on my word, you can download a free trial that seems fully functional, it only shows a trial message when playing a movie.

Linux: script that creates table of network interface properties

2013-08-15T00:00:00+02:00

My server has 5 network interfaces and I wanted a quick overview of some properties. There may be an existing linux command for this but I couldn't find it so I quickly wrote my own script (download).

This is the output:

The only requirement for this script is that you have 'ethtool' installed.

Update 2013-08-17

I recreated the script in python (download) so I can just dynamically format the table and not use ugly hacks I used in the bash script.

How to compile HAProxy from source and setup a basic configuration

2013-08-14T01:00:00+02:00

To learn more about HAProxy I decided to compile it from source and use it to load-balance traffic to louwrentius.com across two different web servers.

I run HAProxy on a VPS based on Ubuntu 12.04 LTS. Let's dive right in.

First, we need to download the source. Don't copy/pased the exact code, you should download the latest version of HAProxy.

cd /usr/src
wget "http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.24.tar.gz"
tar xzf haproxy-1.4.24.tar.gz
cd haproxy-1.4.24

Before you can compile software, you must make sure you have a working build-environment. With Ubuntu or Debian, you should run:

apt-get install build-essential

If you open the README file in the root directory, you will find some detailed instructions on how to compile HAProxy, which is really straight-forward.

Compiling HAProxy

Best CPU performance

The manual states that by default, it will compile HAProxy with no CPU-specific optimisations. To enable CPU-specific optimisations, you need to use the 'native' option.