1. Fio-Plot: Creating Nice Charts From FIO Storage Benchmark Data

    November 28, 2019

    New release of fio-plot

    I've rewritten fio-plot, a tool that turns FIO benchmark data into nice charts. It allows you to create four types of graphs which will be discussed below.

    The github project page explains how to run this tool.

    Fio-plot also includes a benchmark script that automates testing with Fio.

    The git repository also contains benchmark data, which can be used to test fio-plot.

    2D bar chart

    This chart plots IOPs and latency for various queue depths.

    2dbar

    3D bar chart

    3dbar

    2D line chart for log data

    2dline

    2D bar chart latency histogram

    histogram

    Additional images

    The github repository also contains a folder with a lot of example images.

    Tagged as : storage
  2. My Home Lab Server With 20 Cores / 40 Threads and 128 GB Memory

    August 13, 2019

    dl380gen8

    I've bought a secondhand HP DL380p Gen8 server and I think it's awesome.

    This machine has dual-processors, with each CPU having 10 physical cores and two threads per core, for a grand total of 20 cores / 40 threads for the entire machine. It's also equiped with 128 GB or memory.

    htop

    Laughs in htop

    I bought this machine because I wanted to have a dedicated server on which I run my home lab environment. This is the box on which I will try out new software, run experiments, test ideas and so on.

    In this article I'd like to share some information about this machine and my experiences with it. Let's start with an overview of the specifications.

    Update: this article featured on Hacker News, click here for the discussion thread.

    HP DL380p Gen8 Specifications

    PartDescription
    Form factor19" 2U rackmount
    Processor2 x Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
    RAM128 GB DDR3 (16 x 8GB)
    Onboard LAN4 x Broadcom 1 Gigabit (Copper)
    RAID Controller HP P420i
    Storage 8 x 2.5" SAS/SATA slots (no media)
    PSU1 x 750 Watt
    Power usageabout   110 Watt idle
    KVMHP iLO 4 with HTML5 console

    Cost

    I've bought this server for around 1500 Euro ($1700) including taxes (21%) and a one year warranty. I've bought it from the company https://creoserver.com (The Netherlands).

    I believe that the price/performance balance for this server is quite good, but I let you be the judge of that.

    The storage is based on some SSDs I already owned and some new SSDs I've bought separately. So the price is based on the chassis + storage caddies but without actual storage media.

    Age

    The HP DL380p Gen8 product generation was introduced in 2012. The configuration I have chosen is based on a CPU that was introduced in 2013. Based on the serial number I guess the chassis+motherboard is from 2014 (about 5 years old1 at the time I bought it).

    CPU - The Intel Xeon E5-2680 v2

    The Intel Xeon E5-2680 v2 CPU originated from late 2013 and is based on the ivy bridge architecture. The base clock is 2.8 Ghz but it can turbo boost up to 3.6 Ghz. With 10C/20T per CPU and the dual-processor configuration it's needless to say that you can run quite a few virtual machines and/or containers in parallel on this machine, without even oversubscribing CPU cores.

    You may ask what kind of performance you may expect from this box and how 2013 holds up in 2019. A high number of cores is nice, but the cores themselves fast enough?

    The 2680v2 is a processor that sold for about $1700 (Intel Ark) when it came out in 2013. If you look at the list of Xeon processors from that timeframe, it was the top of the line Intel dual-processor compatible CPU with only the E5-2690 v2 processor above it, which ran only 200 Mhz faster (base clock).

    There was almost nothing better at that time and people ran their companies on those machines so I think performance is more than enough. Especially for a lab environment (overkill is probably an understatement).

    I've included some benchmark results below just for reference. You can use this information to make some comparisons to other CPUs that you are familiar with to get some sense of the performance.

    I've also compared scores a little bit and I've noticed that people in the past were perfectly happy gaming on desktop systems with CPUs that have equal or less single core performance.

    Geekbench 4 score

    I've run Geekbench on the DL380p Gen8:

    Single Core:  3525
    Multi Core : 48764
    

    Passmark

    (lookup from this table)

    Single Core:  1824
    Multi Core : 15885
    

    Sysbench

    root@dora:/home/ansible# sysbench --test=cpu --cpu-max-prime=20000 run
    [.....]
    General statistics:
        total time:                          10.0002s
        total number of events:              4358
    [.....]
    

    Some perspective: My intel Mac mini 2018 and my game pc from 2013 also hit 10 seconds runtime. For whoever thinks they should work with 'cheap' Raspberry Pi's, this is the sysbench score of a Raspberry Pi 3B+.

    Test execution summary:
        total time:                          328.3016s
    

    It's 32x slower. I really recommend to not use Pi's except for electronic projects. The performance is so bad, it's not worth it. Even a small VM on your desktop/laptop is already way faster.

    Other benchmarks

    I won't promise anything, but if it's not too much work I can run a benchmark if you have any suggestions. Feel free to comment down below or email me.

    Memory

    I've ordered this machine with 128GB of DDR3 memory. The memory consists of 16 x 8GB 1333Mhz memory modules. The machine has 12 memory slots per CPU so there are still 8 slots left to be filled if I ever feel a need to expand memory.

    Storage

    Drive slots

    I've chosen a chassis with room for 8 x 2.5" drives. Depending on the chassis model, you can go up to 25 x 2.5" (Small Form Factor) or 12 x 3.5" (Large Form Factor).

    Both SAS and SATA HDDs/SSDs are supported. You can't just stick HDDs or SSDs in the drive slots, you need to put your HDD/SSD first in an empty caddy, I bought those with the server.

    caddy

    The caddy has the LED lights build-in to signal the drive status. It has some electrical contacts at the back for those lights, it's not just a plastic 'sabot'.

    My caddies tended to be a bit fragile, they came apart when trying to pull a drive out of the cage, but nothing broke and it all works fine.

    Drive compatibility

    HP servers can be picky about which HDD disk drives work or won't work. They all technically work, but sometimes the disk temperature cannot be read by the controller. In that case, the system fans will slowly start to spin faster and faster until almost maximum speed. This is obviously not workable.

    Smaller HDD drives did work fine and the server stayed quiet. Only some 500GB HDD drives caused 'thermal runaway'.

    I've found this table of supported and unsupported drives for HP controllers.

    Overview of SSDs that are compatible

    I've tested these SSDs:

    - Samsung 860 Pro (consumer)
    - Samsung PM833   (enterprise)
    - Intel D3-S4610  (enterprise)
    - Crucial MX200   (consumer)
    

    P420i RAID mode vs. HBA mode

    Important: you can't boot from the storage controller in HBA mode. You are left with booting from internal/external USB (2.0) or internal SD card.

    I'm not sure if there is any significant performance difference between the two modes for SSDs. I just want to be able to boot from one of the drives, not from USB/SD card.

    Storage management - ssacli tool

    To configure the P420i RAID controller, It's highly recommended to install the ssacli tool from HP. Otherwise you have to reboot and wait 10 minutes (no joke) to enter the 'Smart Array Configuration Utility' to make changes to the storage configuration.

    I just followed these instructions to install this tool on Ubuntu:

    curl http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | apt-key add -
    curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | apt-key add -
    curl http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | apt-key add -
    curl http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | apt-key add -
    

    Contents of /etc/apt/sources.list.d/hp.list:

    deb http://downloads.linux.hpe.com/SDR/repo/mcp/Ubuntu bionic/current non-free
    

    Installing the software:

    apt-get update
    apt-get install ssacli
    

    Some examples:

    ssacli ctrl all show detail
    ssacli ctrl slot=0 array all show detail
    ssacli ctrl slot=0 ld all show detail
    ssacli ctrl slot=0 pd all show detail
    
    ssacli ctrl slot=0 ld 7 delete
    ssacli ctrl slot=0 create type=ld raid=0 drives=2I:2:7 ss=8 ssdopo=off
    
    ssacli ctrl slot=0 array i modify ssdsmartpath=disable
    ssacli ctrl slot=0 pd 1 modify led=on
    

    RAID vs non-RAID

    You are forced by the RAID controller to always create a RAID array, even if you don't need the redundancy or performance.

    I've chosen to just put SSDs in individual RAID0 logical drives, making them effectively individual drives from the perspective of the operating system. I'm not running a mission-critical application so this is fine for me.

    frontside

    A small downside is that the bright red square light on each drive will permanently glow up. That light signals truthfully that you cannot remove a drive without killing the 'array'. I've not seen any option to turn this off, which I can understand.

    Sound / Noise level

    Idle sound level

    I've taken some sound measurements using my iPhone with an app, so how reliable this is, I don't know but it's something.

    During boot time (full fan speed)  : 62 dB
    Idle, booted into operating system : 50 dB
    

    To set some expectations:

    My very subjective opinion is that at 50 dB the sound level is reasonable for a server like this, but it's definitely not quiet. I would not be able to work, relax or sleep with this server in the same room.

    Although this server is fairly quiet at idle, it does need its own dedicated room. When the door is closed, you won't hear it at idle, but under sustained load, you will hear high pitched fan noise even with a closed door.

    Impact of adding a PCIe card on noise level

    When you put any kind of PCIe card in the server, two of its six fans in-line with the PCIe expansion slot will run at 40%+ to cool these cards.

    The sound level will become a bit annoying. I've not found any option to disable this behavior. This means that if you need to expand the server with additional network ports or other components, the server really needs a room with a closed door.

    Please note that if you want to keep noise levels down but want to upgrade to 10Gbit networking, you could opt to configure the server with Flexlom 2 x 10Gbit instead of the stock Flexlom 4 x 1 Gbit copper. This will give you more bandwidth without the need to add a PCIe card.

    At this time, I have no need to add any high-speed networking as I can run the simulations all on the machine itself (at least that's the plan).

    Idle power usage

    The DL380p has several BIOS settings that trade performance for (idle) power consumption. I have tested the performance and idle power usage for two power profiles:

    HP Power Profile: Balanced Power and Performance
    Present Power Reading: 76 Watts
    Geekbench Single Core score: 2836
    
    HP Power Profile: Static High Performance
    Present Power Reading:  100 Watts
    Geekbench Single Core score: 3525
    

    Please note: that actual power usage includes +10 Watt for the iLO interface, to the system would use 86 Watt or 110 Watt at the wall outlet.

    What we can learn from the above test results is that the high performance setting uses 31% more power at idle and in return you get 20% more single core performance.

    The server supports (obviously) dual power supplies but that option only adds cost, increases power usage and gains me nothing. I'm not running a mission-critical business with this server.

    By default, this server is turned off. When I want to do some work, I turn the server on using wake-on-lan. I do this with all my equipment except for my router/firewall.

    Boot up time

    I clocked 4 minutes and 28 seconds until I got a ping reply from the operating system.

    KVM / iLO (Remote Management)

    As with almost all HP servers, this machine has a dedicated remote management port + engine (iLO) so you can do everything remotely, such as powering the server on/off. You also have access to a virtual console (KVM over IP) and virtual CD-ROM drive to remotely provision the server.

    backside

    For those of you who are less familiar with this hardware: notice the network interface left of the VGA connector. This is the iLO remote management interface. It's a mini computer with its own operatingsystem and IP-address, thus reachable from the network.

    In the past iLO or similar solutions were a pain to use because you need Java or .NET to use the virtual console. With the ilO 2.7 update you will have a HTML5 interace, doing away with the need for both Java as .NET. This is a huge usability improvement.

    Operating system

    I'm currently running Ubuntu Linux 18.04 and using plain vanilla KVM to spawn virtual machines, wich works fine. Everything is maintained/configured with Ansible.

    Closing words

    I hope this overview was informative. Maybe it could be an option for yourself to consider if you ever want to setup a home lab yourself.


    1. As far as I know this is not 100% reliable as the motherboard serial number can be changed to match the chassis when it's replaced due to failure. I guess it will have to do. 

    Tagged as : Server
  3. My Ceph Test Cluster Based on Raspberry Pi's and HP MicroServers

    January 27, 2019

    Introduction

    To learn more about Ceph, I've build myself a Ceph Cluster based on actual hardware. In this blogpost I'll discus the cluster in more detail and I've also included (fio) benchmark results.

    This is my test Ceph cluster:

    picluster

    The cluster consists of the following components:

     3 x Raspberry Pi 3 Model B+ as Ceph monitors
     4 x HP MicroServer as OSD nodes (3 x Gen8 + 1 x Gen10)
     4 x 4 x 1 TB drives for storage (16 TB raw)
     3 x 1 x 250 GB SSD (750 GB raw)
     2 x 5-port Netgear switches for Ceph backend network (bonding)
    

    Monitors: Raspberry Pi 3 Model B+

    I've done some work getting Ceph compiled on a Raspberry Pi 3 Model B+ running Raspbian. I'm using three Raspberry Pi's as Ceph monitor nodes. The Pi boards don't break a sweat with this small cluster setup.

    Note: Raspberry Pi's are not an ideal choice as a monitor node because Ceph Monitors write data (probably the cluster state) to disk every few seconds. This will wear out the SD card eventually.

    Storage nodes: HP MicroServer

    The storage nodes are based on four HP MicroServers. I really like these small boxes, they are sturdy, contain server-grade components, including ECC-memory and have room for four internal 3.5" hard drives. You can also install 2.5" hard drives or SSDs.

    For more info on the Gen8 and the Gen10 click on their links.

    Unfortunately the Gen8 servers are no longer made. The replacement, the Gen10 model, lacks IPMI/iLO and is also much more expensive (in Europe at least).

    CPU and RAM

    All HP Microservers have a dual-core CPU. The Gen8 servers have 10GB RAM and the Gen10 server has 12GB RAM. I've just added an 8GB ECC memory module to each server, the Gen10 comes with 4GB and the Gen8 came with only 2GB, which explains the difference.

    Boot drive

    The systems all have an (old) internal 2.5" laptop HDD connected to the internal USB 2.0 header using an USB enclosure.

    Ceph OSD HDD

    All servers are fitted with four (old) 1TB 7200 RPM 3.5" hard drives, so the entire cluster contains 16 x 1TB drives.

    Ceph OSD SSD

    There is a fifth SATA connector on the motherboard, meant for an optional optical drive, which I have no use for and wich is not included with the servers.

    I use this SATA connector in the Gen8 MicroServers to attach a Crucial 250GB SSD, which is then tucked away at the top, where the optical drive would sit. So the Gen8 servers have an SSD installed which the Gen10 is lacking.

    The entire cluster thus has 3 x 250GB SSDs installed.

    Networking

    All servers have two 1Gbit network cards on-board and a third one installed in one of the half-height PCIe slots1.

    backside

    The half-height PCIe NICs connect the Microservers to the public network. The internal gigabit NICs are configured in a bond (round-robin) and connected to two 5-port Netgear gigabit switches. This is the cluster network or the backend network Ceph uses for replicating data between the storage nodes.

    You may notice that the first onboard NIC of each server is connected to the top switch and the second one is connected to the bottom switch. This is necessary because linux round-robin bonding requires either separate VLANs for each NIC or in this case separate switches.

    Benchmarks

    Benchmark conditions

    • The tests ran on a physical Ceph client based on an older dual-core CPU and 8GB of RAM. This machine was connected to the cluster with a single gigabit network card.
    • I've mapped RBD block devices from the HDD pool and the SSD pool on this machine for benchmarking.
    • All tests have been performed on the raw /dev/rbd0 device, not on any file or filesystem.
    • The pools use replication with a minimal copy count of 1 and a maximum of 3.
    • All benchmarks have been performed with FIO.
    • All benchmarks used random 4K reads/writes
        NAME     ID     USED        %USED     MAX AVAIL     OBJECTS
        hdd      36     1.47TiB     22.64       5.03TiB      396434
        ssd      38      200GiB     90.92       20.0GiB       51204
    

    Benchmark SSD

    Click on the images below to see a larger version.

    a b c d e f

    Benchmark HDD

    g h i j k l

    Benchmark evaluation

    The random read performance of the hard drives seems unrealistic at higher queue depths and number of simultaneous jobs. This performance cannot be sustained purely on the basis that 16 hard drives with maybe 70 random IOPs each can only sustain 1120 random IOPs.

    I cannot explain why I get these numbers. If anybody has a suggestion, feel free to comment/respond. Maybe the total of 42GB of memory across the cluster may act as some kind of cache.

    Another interesting observation is that a low number of threads and a small IO queue depth results in fairly poor performance, both for SSD and HDD media.

    Especially the performance of the SSD pool is poor with a low IO queue depth. A probable cause is that these SSDs are consumer-grade and don't perform well with low queue depth workloads.

    I find it interesting that even over a single 1Gbit link, the SSD-backed pool is able to sustain 20K+ IOPs at higher queue depths and larger number of threads.

    The small number of storage nodes and the low number of OSDs per node doesn't make this setup ideal but it does seem to perform fairly decent, considering the hardware involved.


    1. You may notice that the Pi's are missing in the picture because this is an older picture when I was running the monitors as virtual machines on hardware not seen in the picture. 

    Tagged as : Ceph

Page 1 / 65