Louwrentius

In my last blog post I discussed how a small solar project - to power this blog on a Raspberry Pi - escalated into a full-blown off-grid solar setup, large enough to power the computer I use at the moment to write this update¹. In this update, I want to discuss my battery upgrade.

For me, the huge lead acid battery (as pictured below) was always a relatively cheap temporary solution.

A 12 Volt 230 Ah lead-acid battery

Lead-acid batteries are not ideal for solar setups for multiple reasons, but the most problematic issue is the slow charging speed. I only have a few hours of direct sunlight per day due to my particular situation and the battery just could not absorb sunlight fast enough.

For the last 5-7 years, the go-to battery chemistry for solar is LiFePO4 or lithium iron phosphate as a replacement for lead-acid batteries. This battery chemistry is not as energy-dense as Lithium-ion, but the upside is price and safety. In particular, LiFePO4 cells aren't as volatile as Lithium-ion cells. They may start outgassing, but they don't start a fire.

More importantly for my situation: LiFePO4 batteries can charge and discharge at much higer rates than lead-acid batteries². It's possible to charge LiFePO4 cells with a C-rate of 1! This means that if a battery is rated for 100Ah (Ampere-hours) you can charge with a current of 100 Ampere! My solar setup will never come even close to that number, but at least it's good to have some headroom.

A single 3.2 volt 230Ah Lithium Iron Phosphate prismatic cell

I did contemplate buying an off-the-shelf battery but I decided against it. You have no control over the brand and quality of the LiFePO4 cells they use and more importantly, what's the fun in that anyway?

So I decided to order my own cells and build my own 12 Volt LiFePO4 battery consisting of four cells in series (4S) as my existing system is also based on 12 Volt. Other common configurations are 8S (24 Volt) and 16S (48 Volt³).

It turned out that I could just buy my cells locally in The Netherlands (instead of China) because of a company that specializes in batteries (no affiliate). As the price was right, I bought effectively 3 KWh for just shy of 500 Euros.

I decided to buy B-grade cells as those are cheaper than A(utomotive)-grade cells. I might have gone for A-grade cells as not to risk anything if I would build a more serious battery bank for my whole home. Yet a lot of people report no significant differences between A-grade and B-grade LiFePO4 cells for solar battery banks so in the end, it's all about your particular apetite for risk.

Just buying cells and putting them in series (in my case 4S) is not enough, a BMS or battery management system is needed, which you put in series with the battery on the negative terminal. I ordered a 100A Daly BMS from China which works fine. I'm even able to use Python to talk with the Daly BMS over bluetooth to extract data (voltages, current, State of Charge and so on).

The BMS is critical because it protects the cells against deep discharge and overcharging. In addition, the BMS tries to keep the voltage of the cells as equal as possible, which is called 'balancing'. Charging stops entirely when just one of the cells reach their maximum voltage. If other cells have a much lower voltage, it means that they can still be charged but the one cell with the high voltage is blocking them from doing so. That's why cell balancing is critical if you want to use as much of the capacity as possible.

The Daly BMS is quite bad at cell balancing so I've ordered a separate cell balancer for $18 to improve cell balancing (yet to be installed).

Ikea sells a Kuggis 32cmx32cmx32cm storage box that seems to be perfect for my small battery. As it has two holes on the sides, I just routed the positive and negative cables through them.

Now that I've put this battery in place I've seen a huge improvement regarding solar charge performance.

I've actually potentially created a new problem: my solar charge controller can only handle about 400 Watts of solar power at 12V and my setup is quite close to reaching this output. I may have undersized my solar charge controller and it has come back to bite me. For now, I'm going to just observe: if that peak of 400 Watts is only reached for a brief time - as it is right now - I don't think I'm going to upgrade my solar charge controller as that would not be worth it.

As we are still in May, my best yield is 1.2 KWh per day. Although that's paltry as compared to regular residential solar setups, that 1.2 KWh is more than a third of my battery capacity and can run my computer setup for 10 hours, so for me it's good enough.

It's funny to me that all of this started out with just a 60 Watt solar panel, a 20 Euro solar charge controller (non MPPT) and a few 12V 7Ah lead acid gel batteries in parallel.

I think it's beyond amazing that you can now build a 15KWh battery bank complete with BMS for less than €3000. For that amount of money, you can't come even close to this kind of capacity.

For context, it's also good to know that the longevity of LiFePO4 cells is amazing. A-grade cells are rated for 6000 cycles ( 16+ years at one cycle per day ) and my vendor rated B-grade cells at 4000 cycles (~11 years).

Maybe my battery build may inspire you to explore building your own battery. LiFePO4 cells come in a whole range of capacities, I've seen small 22Ah cells or huge 304Ah cells so you can select something that fits your need and budget.

If you're looking for more information: there are quite a few Youtubers that specialise in building large battery banks (48 Volt, 300Ah, ~15KWh) to power their homes and garages.

Although Will Prowse reviewed LiFePO4 cells in the past, he currently focusses mostly on off-the-shelf products, like "rack-mount" batteries and inverter/chargers.

I also like the off-grid-garage channel a lot, the channel as tested and explored quite a few products.

Harrold Halewijn (Dutch) also has quite a few videos about solar setups in general and solar battery setups. He's really into automation, in combination with flexible (next-day) energy prices.

Also in Dutch, a cool article about some people building their own large-scale home storage batteries (15KWh+)

Another Dutch person build a solar power factory with a battery capacity of 128 KWh for professional energy production. Truely amazing.

The Hacker News thread about this article.

In 2020 I wondered if I could run my blog on solar power, being inspired by Low-tech Magazine, doing the same thing (but better)¹. The answer was 'yes', but only through spring and summer.

I live in an apartment complex in The Netherlands and my balcony is facing west. This means it only receives direct sunlight from 16:00 onward during spring and summer. Most of the time, the panels only get indirect sunlight and therefore generate just a tiny fraction of their rated performance. The key issue is not solar, but the west-facing balcony (it should ideally be facing south).

original solar panel

It's fair to say that my experiment isn't rational because of the sub-optimal solar conditions. Yet, I'm unreasonably obsessed by solar power and I wanted to make it work, even if it didn't make sense from an economic or environmental perspective².

When I wrote my blog about my solar-powered setup, I was already on my second iteration: I started out with just a 60 Watt panel and a cheap $20 solar controller³. That didn't even come close to being sufficient, so I upgraded the solar controller and bought a second panel rated for 150 Watt, which is pictured above. With the 60 Watt and 150 Watt panels in parallel, it was still not enough to keep the batteries charged in the fall and winter, due to the west-facing balcony.

A Raspberry Pi 4B+ consumes around ~3.5 Watt of power continuously. Although that sounds like a very light load, if you run it for 24 hours, it's equivalent to using 84 Watts continuously for one hour. That's like running two 40 Watt fans for one hour, it's not insignificant and it doesn't even account for battery charging losses.

So 210 Watt of solar (receiving mostly indirect sunlight) still could not power my Raspberry Pi through the winter under my circumstances. Yet, in the summer, I had plenty of power available and had no problems charging my iPad and other devices.

As my solar setup could not keep the batteries charged from October onward, I decided to do something radical. I bought a 370 Watt⁴ solar panel (1690 x 1029 mm) and build a frame made of aluminium tubing⁵. Solar panels have become so cheap that the aluminium frame is more expensive than the panel.

Even this 370 Watt panel was not enough during the gloomy, cloudy winter days. So I bought a second panel and build a second tube frame. Only with a 740 Watt rated solar panel setup was I able to power my Raspberry Pi through the winter⁶.

I didn't create this over-powered setup just to power the Raspberry Pi during the winter. I knew that solar performed much better during spring and summer and I wanted to capture as much of that energy as possible. The real goal was to go beyond powering the Pi and power my computer desk, which includes an Intel Mac Mini, two 1440p 27" displays and some other components (using around 100 Watt on average)⁷.

I would not be able to power my desk 24/7 but I would be happy if I can work on solar power for a few hours every other day during spring and summer. I also wanted to light my house in the evening using this setup.

The original solar setup was enough to power the Raspberry Pi and charge an iPad in the spring/summer. The solar charge controller could not handle the increased solar capacity and needed replacement. So I decided to build a new setup inspired by Will Prowse solar demo setups, which is pictured below:

the latest iteration of my solar setup

First a brief disclaimer: I'm a hobbyist, not an expert (if you didn't notice already). I have no background in electrical systems. I've tried to make my setup safe, but I may have done things that are not recommended.

My setup is a 12-volt system⁹. The drawback of a 12-volt system is the relatively large currents required to charge the battery and power the inverter. This requires thicker, more expensive cabling to prevent energy losses in the cabling⁸.

Most components are self-explanatory, except for the shunt. This device precisely measures battery voltage and how much current is going in and out of the battery. The solar charge controller and the shunt are linked together in a bluetooth network, so the solar controller uses the precise voltage and current information from the shunt to regulate the battery charging process.

The solar controller, inverter and shunt have VE.direct interfaces (Victron-specific) which I use to collect data. I'm using a Python VE.direct module to gather this data, which just works without any issues. My Python script dumps the data into InfluxDB and I use Grafana for graphs (see below). The script also updates the 'solar status' bar to the right (or bottom for mobile users).

this image is updated periodically

The LCD display is just for fun, and mostly to keep an eye on the battery charge state.

the 20x4 LCD screen

The LCD screen is managed by the same python script that dumps the VE.direct data into Grafana. I focus on two metrics in particular. First of all the daily solar yield as a percentage: 100% means the load has been compensated by solar and anything higher means an energy 'profit'. In the bottom right we see the charger status (Bulk): if it's on 'Float' the battery is full. I tend to wait for the battery to recharge to a 'float' status before I use the inverter again.

Let's talk about the battery. I've chosen to use a large used lead-acid battery even though Lithium (LiFePO4) batteries beat lead-acid in every metric.

A 12 Volt 230 Ah lead-acid battery¹⁰

I bought the battery¹¹ second-hand for €100 so that's not a significant investment for a battery. Although it's a bit worn-down and the capacity is reduced, it is still good enough for me to run my computer setup for 10 hours after a full charge¹². In practice, I won't use the battery for more than four to five hours at a time because recharging can take multiple days and lead-acid batteries should ideally be fully recharged within 24 hours or their aging is accelerated.

The lead-acid battery also serves another purpose: is a relatively cheap option for me to validate my setup. If it works as intended, I might opt to upgrade to lithium (LiFePO4) at some point.

Until recently, switching between the grid and solar for my computer setup was quite cumbersome. I had to power down all equipment, connect to the inverter and power everything up again. That got old very quickly. Fortunately, I stumbled on an advertisement for a Victron Filax 2 and it turns out that it does exactly what I need.

The Filax 2 switches between two 230 Volt input sources without any interruption, like a UPS (Uninterruptible power supply). Now that I've installed this device, I can switch between solar and grid power without any interruption. Brand new, The Filax 2 costs €350 which was beyond what I wanted to spend, but the second-hand price was acceptable.

My solar setup is not something that I can just turn on and forget about. I have to keep an eye on the battery, especially because lead-acid should ideally be recharged within 24 hours.

Happy case: the battery is full and my computer desk is 100% solar-powered

It's now April 2023 and my setup seems promising. Peak output of the two 370 Watt solar panels facing west was 230 Watts. Only for a very short period, but it makes me confident for spring and summer. I could automate enabling and disabling the inverter, with a relay and some logic in Python, but for now I'm good with manually operating the inverter.

You may have noticed that I've used a lot of Victron equipment¹³. Mostly because it seems high-quality and the data interfaces are documented and easy-to-use. The inverter was also chosen because of the low parasitic load (self-consumption) of around 6 watt. Victron equipment is not cheap. Buying Victron gear second-hand can save a lot of money.

Speaking of cost, if I include all the cost I made, including previous solar projects and mistakes, I think I spend around €2000.

That's all I have to say about my hobby solar project for now.

Link to Hacker News thread about this article.

The price of Solid-state drives (SSDs) has dropped significantly over the last few years. It's now possible to buy a 1TB solid-state drive for less than €60. However, at such low price points, there is a catch.

Although cheap SSDs do perform fine regarding reads, sustained write performance can be really atrocious. To demonstrate this concept, I bought a bunch of the cheapest SATA SSDs I could find - as listed below - and benchmarked them with Fio.

I didn't have the budget to buy a bunch of 1TB of 2TB SSD, so these ultra-cheap, low capacity SSDs are a bit of a stand-in. I've also added a Crucial MX500 1TB (CT1000MX500SSD1) SATA¹ SSD - which I already owned - to the benchmarks to see how well those small-capacity SSDs stack up to a cheap SSD with a much larger capacity.

Understanding SSD write performance

To understand the benchmark results a bit better, we discuss some SSD concepts in this section. Feel free to skip to the actual benchmarks if you're already familiar with them.

SLC Cache

SSDs originally used single-level cell (SLC) flash memory, which can hold a single bit and is the fastest and most reliable flash memory available. Unfortunately, it's also the most expensive. To reduce cost, multi-level cell (MLC) flash was invented, which can hold two bits instead of one, at the cost of speed and longevity². This is even more so for triple-level cell (TLC) and quad-level cell (QLC) flash memory. All 'cheap' SSDs I benchmark use 3D v-nand³ TLC flash memory.

One technique to temporarily boost SSD performance is to use a (small) portion of (in our case) TLC flash memory as if it was SLC memory. This SLC memory then acts as a fast write cache⁴. When the SSD is idle, data is moved from the SLC cache to the TLC flash memory in the background. However, this process is limited by the speed of the 'slower' TLC flash memory and can take a while to complete.

While this trick with SLC memory works well for brief, intermittent write loads, sustained write loads will fill up the SLC cache and cause a significant drop in performance as the SSD is forced to write data into slower TLC memory.

DRAM cache

As flash memory has a limited lifespan and can only take a limited number of writes, a wear-leveling mechanism is used to distribute writes over all cells evenly, regardless of where data is written logically. Keeping track of this mapping between logical and physical 'locations' can be sped up with a DRAM cache (chip) as DRAM tend to be faster than flash memory. In addition, the DRAM can also be used to cache writes, improving performance. Cheap SSDs don't use DRAM cache chips to reduce cost, thus they have to update their data mapping tables in flash memory, which is slower. This can also impact (sustained) write performance. To be frank, I'm not sure how much a lack of DRAM impacts our benchmarks.

Benchmark method

Before I started benchmarking I submitted a trim command to clear each drive. Next, I performed a sequential write benchmark of the entire SSD with a block size of 1 megabyte and a queue depth of 32. The benchmark is performed on the 'raw' device, no filesystem is used. I used Fio for these benchmarks.

Benchmark results

The chart below shows write bandwith over time for all tested SSDs. Each drive has been benchmarked in full, but the data is truncated to the first 400 seconds for readability (performance didn't change). The raw Fio benchmark data can be found here (.tgz).

click for a larger image

It's funny to me that some cheap SSDs initially perform way better than the more expensive Crucial 1TB SSD⁵. As soon as their SLC cache runs out, the Crucial 1TB has the last laugh as it shows best sustained throughput, beating all cheaper drives, but the Kingston A400 comes close.

Of all the cheap SSDs only the Kingston shows the best sustained write speed at around 100 MB/s and there are no intermittent drops in performance. The ADATA, PNY and Verbatim SSDs show flakey behaviour and basically terrible sustained write performance. But make no mistake, I would not call the performance of the Kingston SSD, nor the Crucial SSD - added as a reference - 'good' by any definition of that word. Even the Kingston can't saturate gigabit Ethernet.

The bandwidth alone doesn't tell the whole story. The latency or responsiveness of the SSDs is also significantly impacted:

click for a larger image

The Crucial 1TB SSD shows best latency overall, followed by the Kingston SSD. The rest of the cheap SSDs show quite high latency spikes and very high latency overall, even when some of the spikes settle, like for the ADATA SSD. When latency is measured in seconds, things are bad.

To put things a bit in perspective, let's compare these results to a Toshiba 8 TB 7200 RPM hard drive I had lying around.

click for a larger image

The hard drive shows better write throughput and latency⁶ as compared to most of the tested SSDs. Yes, except for the initial few minutes where the cheap SSDs tend to be faster (except for the Kingston & Crucial SSDs) but how much does that matter?

As we've shown the performance of a hard drive to contrast the terrible write performance of the cheap SSDs, it's time to also compare them to a more expensive, higher-tier SSD.

click for a larger image

I've bought this Samsung SSD in 2019 for €137 euro, so that's quite a different price point. I think the graph speaks for itself, especially if you consider that this graph is not truncated, this is the full drive write.

Evaluation & conclusion

One of the funnier conclusions to draw is that it's beter to use a hard drive than to use cheap SSDs if you need to ingest a lot of data. Even the Crucial 1TB SSD could not keep up with the HDD.

A more interesting conclusion is that the 1TB SSD didn't perform that much better than the small cheaper SSDs. Or to put it differently: although the performance of the small, cheap SSDs is not representative of the larger SSD, it is still quite in the same ball park. I don't think it's a coincidence that the Kingston SSD came very close to the performance of the Crucial SSD, as it's the most 'expensive' of the cheap drives.

In the end, my intend was to demonstrate with actual benchmarks how cheap SSDs show bad sustained write performance and I think I succeeded. I hope it helps people to understand that good SSD write performance is not a given, especially for cheaper drives.

The Hacker News discussion of this blog post can be found here

Disclaimer

I'm not sponsored in any way. All mentioned products have been bought with my own money.

The graphs are created with fio-plot, a tool I've made and maintain. The benchmarks have been performed with bench-fio, a tool included with fio-plot, to automate benchmarking with Fio.