UPDATE 2019 / 2020
There are official ARM64 binaries of Ceph that you can run on a 64-bit version of Ubuntu 18.04.
Important: I consider this page obsolete. I will keep it up for transparency's sake
Introduction
In this blog post I'll show you how to compile Ceph Luminous for the Raspberry Pi 3B+.
If you follow the instructions below you can compile Ceph on Raspbian. A note of warning: we will compile Ceph on the Raspberry Pi itself which takes a lot of time.
Ubuntu has packages for Ceph on armhf but I was never able to get Ubuntu working properly on the Raspberry Pi 3B+. Maybe that's just me and I did something wrong. Using existing Ceph packages on Ubuntu would probably be the fastest way to get up and running on the Raspberry Pi if it works for you.
This is my test Ceph cluster:
3 x Raspberry Pi 3B+ as Ceph monitors.
4 x HP Microserver as OSD nodes.
4 x 4 x 1 TB drives for storage (16 TB raw)
3 x 1 x 250 GB SSD (750 GB raw)
2 x 5-port Netgear switches for Ceph backend network (bonding)
For the impatient
If you just want the packages you can download this file and you'll get a set of .deb files which you need to install on your Raspberry Pi.
SECURITY WARNING: these packages are created by me, an unknown, untrusted person on the internet. As a general rule you should not download and install these packages as they could be malicious for all you know. If you want to be safe, compile Ceph yourself.
Skip to the section about installing the packages at the end for further installation instructions.
The problem with compiling Ceph for armhf
There are no armhf packages for Ceph because if you try to compile Ceph on armhf the compiler (gcc) will run out of virtual memory (about three gigabytes).
The solution
Daniel Glaser discovered that he could actually compile Ceph on armhf by using Clang/LLVM as the C++ compiler. This compiler seems to use less memory and thus stay within the 3 GB memory boundary. This is why he and I were able to compile Ceph.
How to compile Ceph for armhf - preparation
The challenge: one gigabyte of memory
The Raspberry Pi 3B+ has only one gigabyte of memory but we need more. The only way to add memory is to use swap on disk, as far as I know.
If you use storage as a substitute for RAM memory, you need fast storage, so it's really recommended to use an external SSD drive that you connect through USB. You also may need sufficient storage, I'd recommend 20+ GB.
SD memory cards are not up to the task regarding being used as swap. You'll wear them out prematurely and performance is abysmal. You should really use an external SSD.
Preparing the external SSD
Attach the SSD drive to the Raspberry Pi with USB
- The SSD will probably show up as '/dev/sda'.
- mkfs.xfs /dev/sda -f ( this will erase all contents of the SSD ).
- mkdir /mnt/ssd
- mount /dev/sda /mnt/ssd
Creating and activating swap
- cd /mnt/ssd
- dd if=/dev/zero of=swap.dd bs=1M count=5000
- swapon /mnt/ssd/swap.dd
- swapoff /var/swap
By default, Raspbian configures a 100 MB swap file on /var/swap. In order to increase performance and protect the SD card from wearing out, please don't forget this last step to disable this swap file on the SD card.
Extra software
I would recommend installing 'htop' for real-time monitoring of cpu, memory and swap usage if you like to do so.
- apt-get install htop
How to compile Ceph for armhf - building
Installing an alternative C++ compiler (Clang/LLVM)
As part of Daniel's instructions, you need to compile and install Clang/LLVM. I followed his instructions to the letter, I have not tested the Clang/LLVM packages made available through apt.
Compiling Clang/LLVM takes a lot of time. It took 8 hours to compile LLVM/Clang on a Raspberry Pi 3B+ with make -j3 to limit memory usage.
real 493m38.472s
user 1223m39.063s
sys 45m45.748s
I'll reproduce the steps from Daniel here:
apt update
apt install -y build-essential ca-certificates vim git
apt install libcunit1-dev libcurl4-openssl-dev python-bcrypt python-tox python-coverage
cd /mnt/ssd
mkdir git && cd git
git clone https://github.com/llvm-mirror/llvm.git
cd llvm/tools
git clone https://github.com/llvm-mirror/clang.git
git clone https://github.com/llvm-mirror/lld.git
cd /tmp
mkdir llvm-build && cd llvm-build
cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=ARM /mnt/ssd/git/llvm/
make -j3
make install
update-alternatives --install /usr/bin/cc cc /usr/local/bin/clang 100
update-alternatives --install /usr/bin/c++ c++ /usr/local/bin/clang++ 100
update-alternatives --install /usr/bin/cpp cpp /usr/local/bin/clang-cpp 100
You may chose to build in some other directory, maybe on the SSD itself. I'm not sure if that makes a big difference. Be carefull when using /tmp as all contents are lost after a reboot.
Obtaining Ceph
There are two options: 1. clone my Luminous fork containing the branch 'ceph-on-arm' which incorporates all the 'fixed' files that make Ceph build with Clang/LLVM.
- Clone the official Ceph repo and use the luminous branche. Next, you edit all the relevant files and make the changes yourself. Here you can find a list of all the files and the changes made to them.
I would recommend to just git clone ceph like this:
cd /mnt/ssd
git clone https://github.com/louwrentius/ceph
cd ceph
git checkout ceph-on-arm
git reset --hard
git clean -dxf
git submodule update --init --recursive
Now we first need to install a lot of dependancies on the Raspberry Pi before we can build Ceph.
run ./install-deps.sh
This will take some time as a ton of packages will be installed. Once this is done we are ready to compile Ceph itself.
Building Ceph
So to understand what you are getting into: it took me about 12 hours to compile Ceph on a Raspberry Pi 3 B+
real 717m31.457s
user 1319m50.438s
sys 58m7.549s
This is the command to run:
./make-debs.sh
If you want to monitor cpu and memory usage, you can use 'htop' to do so.
If for some reason the compile proces does fail and you may have to restart compiling ceph after you made some adjustments:
(you may have to adjust the folder name to match your ceph version)
cd /tmp/release/Raspbian/WORKDIR/ceph-12.2.9-39-gd51dfb14f4
< edit the relevant files here >
dpkg-buildpackage -j3 -us -us -nc
Once this process is done, you will find a lot of .deb packages in your /tmp/release/Raspbian/WORKDIR folder.
Warning If you do use /tmp, the first thing to do is to copy all .deb files to a safe location because if you reboot your Pi, you loose 12 hours of work.
Assuming that you copied all .deb files to a folder like '/deb' you just created, this is how you install these packages:
dpkg --install *.deb
apt-get install --fix-missing
apt --fix-broken install
This is a bit ugly but it worked fine for me.
You can now just copy over all the .deb files to other Raspbery Pi's and install Ceph on them too.
Now you are done and you can run Ceph on a Raspberry Pi 3B+.
Ceph monitors may wear out the SD card
Important Running a Ceph monitor node on a Raspberry Pi is not ideal. The core issue is that the Ceph monitor process writes data every few seconds to files within /var/lib/ceph and this may wear out the SD card prematurely. The solution would be to use an external usb hard drive mounted through USB or a regular ssd which is way more resilient to writes than a regular SD card.
Comments