1. Understanding the Ubuntu 20.04 LTS Server Autoinstaller

    Thu 11 February 2021

    Introduction

    Ubuntu Server version 18.04 LTS uses the debian-installer (d-i) for the installation process. This includes support for 'preseeding' to create unattended (automated) installations of ubuntu.

    d-i

    the debian installer

    With the introduction of Ubuntu Server 20.04 'Focal Fossa' LTS, back in April 2020, Canonical decided that the new 'subiquity server installer' was ready to take it's place.

    After the new installer gained support for unattended installation, it was considered ready for release. The unattended installer feature is called 'Autoinstallation'.

    I mostly run Ubuntu 18.04 LTS installations but I decided in February 2021 that I should get more acquainted with 20.04 LTS, especially when I discovered that preseeding would no longer work.

    In this article I assume that the reader is familiar with PXE-based unattended installations.

    Why this new installer?

    Canonical's desire to unify the codebase for Ubuntu Desktop and Server installations seems to be the main driver for this change.

    From my personal perspective, there aren't any new features that benefit my use-cases, but that could be different for others. It's not a ding on the new Autoinstaller, it's just how I look at it.

    There is one conceptual difference between the new installer and preseeding. A preseed file must answer all questions that the installer needs answered. It will switch to interactive mode if a question is not answered, breaking the unattended installation process. From my experience, there is a bit of trial-and-error getting the preseed configuration right.

    The new Subiquity installer users defaults for all installation steps. This means that you can fully automate the installation proces with just a few lines of YAML. You do't need an answer for each step.

    The new installer as other features such as the ability to SSH into an installer session. It works by generating an at-hoc random password on the screen / console which you can use to logon remotely over SSH. I have not used it yet as I never found it necessary.

    Documentation is a bit fragmented

    As I was trying to learn more about the new Autoinstaller, I noticed that there isn't a central location with all relevant (links to) documentation. It took a bit of searching and following links, to collect a set of useful information sources, which I share below.

    Link Description
    Reference Manual Reference for each particular option of the user-data YAML
    Introduction An overview of the new installer with examples
    Autoinstall quick start Example of booting a VM using the installer using KVM
    Netbooting the installer A brief instruction on how to setup PXE + TFTP with dnsmasq in order to PXE boot the new installer
    Call for testing Topic in which people give feedback on their experience with the installer (still active as of February 2021) with a lot of responses
    Stack Exchange post Detailed tutorial with some experiences.
    Medium Article Contains an example and some experiences.
    github examples Github repo with about twenty examples of more complex configurations

    The reference documentation only supports some default use-cases for the unattended installation process. You won't be able to build more complex configuration such as RAID-based installations using this reference.

    Under-the-hood, the installer uses curtin. The linked documentation can help you further build complex installations, such as those which use RAID.

    I think the curtin syntax is a bit tedious and fortunately it is probably not required to learn curtis and piece together more complex configurations by hand. There is a nice quality-of-life feature that takes care of this.

    More on that later.

    How does the new installer work?

    With a regular PXE-based installation, we use the 'netboot' installer, which consists of a Linux kernel and an initrd image (containing the actual installer).

    This package is about 64 Megabytes for Ubuntu 18.04 LTS and it is all you need, assuming that you have already setup a DHCP + TFTP + HTTP environment for a PXE-based installation.

    The new Subiquity installer for Ubuntu 20.04 LTS deprecates this 'netboot' installer. It is not provided anymore. Instead, you have to boot a 'live installer' ISO file which is about 1.1 GB in size.

    The process looks like this:

    1. Download the Live installer ISO
    2. Mount the iso to acquire the 'vmlinuz' and 'initrd' files for the TFTP root
    3. Update your PXE menu (if any) with a stanza like this:
    LABEL focal2004-preseed
                    MENU LABEL Focal 20.04 LTS x64 Manual Install
                    KERNEL linux/ubuntu/focal/vmlinuz
                    INITRD linux/ubuntu/focal/initrd
                    APPEND root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=http://10.10.11.1/ubuntu-20.04.1-live-server-amd64.iso
    

    This process is documented here with detailed how-to steps and commands.

    We have not yet discussed the actual automation part, but first we must address a caveat.

    The new installer requires 3GB of RAM when PXE-booting

    Although it is not explicitly documented1, the new mechanism of PXE-booting the new Ubuntu installer using the Live ISO requires a minimum of 3072 MB of memory. And I assure you, 3000 MB is not enough.

    It seems that the ISO file is copied into memory over the network and extracted on the RAM disk. With a RAM disk of 1500 MB and an ISO file of 1100 MB we are left with maybe 472 MB of RAM for the running kernel and initrd image.

    To put this into perspective: I could perform an unattended install of Ubuntu 18.04 LTS with only 512 MB of RAM2.

    Due to this 'new' installation process, Ubuntu Server 20.04 has much more demanding minimum system requirements than Windows 2019 Server, which is fine with 'only' 512 MB of RAM, even during installation. I have to admit I find this observation a bit funny.

    It seems that this 3 GB memory requirement for the installation process is purely and solely because of the new installation process. Obviously Ubuntu 20.04 can run in a smaller memory footprint once installed.

    Under the hood, a tool called 'casper' is used to bootstrap the installation process and that tool only supports a local file system (in this case on a RAM disk). On paper, casper does support installations using NFS or CIFS but that is not supported nor tested. From what I read, some people tried to use it but it didn't work out.

    As I undertand it, the curent status is that you can't install Ubuntu Server 20.04 LTS on any hardware with less than 3GB of memory using PXE-boot. This probably affects older and less potent hardware, but I can remember a time that this was actually part of the point of running Linux.

    It just feels wrong conceptually, that a PXE-based server installer requires 3GB of memory.


    Important

    The 3GB memory requirement is only valid for PXE-based installations. If you boot from ISO / USB stick you can install Ubuntu Server on a system with less memory. I've verified this with a system with only 1 GB of memory.


    The Autoinstall configuration

    Now let's go back to the actual automation part of Autoinstaller. If we would want to automate our installation, our PXE menu item should be expanded like this:

    APPEND root=/dev/ram0 ramdisk_size=1500000 ip=dhcp url=http://10.10.11.1/ubuntu-20.04.1-live-server-amd64.iso autoinstall ds=nocloud-net;s=http://10.10.11.1/preseed/cloud-init/
    

    In this case, the cloud-init folder - as exposed through an HTTP-server - must contain two files:

    • meta-data
    • user-data

    The meta-data file contains just one line:

    instance-id: focal-autoinstall
    

    The user-data file is the equivalent of a preseed file, but it is YAML-based instead of just regular plain text.

    Minimum working configuration

    According to the documentation, this is a minimum viable configuration for the user-data file:

    #cloud-config
    autoinstall:
      version: 1
      identity:
        hostname: ubuntu-server
        password: "$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0"
        username: ubuntu
    

    This works fine and performs a basic install + apt upgrade of the new system with a disk configuration based on an LVM layout.

    My preferred minimum configuration:

    Personally, I like to keep the unattended installation as simple as possible. I use Ansible to do the actual system configuration, so the unattended installation process only has to setup a minimum viable configuration.

    I like to configure the following parameters:

    • Inject a SSH public key into the authorized_keys file for Ansible
    • Configure the Apt settings to specify which repository to use during installation (I run a local debian/ubuntu mirror)
    • Update to the latest packages during installation

    I'll give examples of the required YAML to achieve this with the new installer.

    Injecting a public SSH key for the default user:

    ssh:
        authorized-keys: |
           ssh-rsa <PUB KEY>
        install-server: true
        allow-pw: no
    

    Notice that we also disable password-authentication for SSH access.

    Configure APT during installation

    I used this configuration to specify a particular mirror for both the installation process and for the system itself post-installation.

    Mirror:
      mirror: "http://mirror.mynetwork.loc"
    apt:
      preserve_sources_list: false
      primary:
        - arches: [amd64]
          uri: "http://mirror.mynetwork.loc/ubuntu"
    

    Performing an apt upgrade

    By default, the installer seems to install security updates but it doesn't install the latest version of softare. This is a deviation from the d-i installer which always ends up with a fully up-to-date system, when done.

    The same end-result can be accomplished by running an apt update and apt upgrade at the end of the installation process.

    late-commands:
    - curtin in-target --target=/target -- apt update           
    - curtin in-target --target=/target -- apt upgrade -y
    

    So all in all, this is not a big deal.

    Network configuration

    The network section can be configured using regular Netplan syntax. An example:

    network:
      version: 2
      renderer: networkd
      ethernets:
        enp0s3:
          dhcp4: no
          addresses:
            - 10.10.50.200/24
          gateway4: 10.10.50.1
          nameservers:
            search:
              - mynetwork.loc
            addresses:
              - 10.10.50.53
              - 10.10.51.53
    

    Storage configuration

    The installer only supports a 'direct' or 'lvm' layout. It also selects the largest drive in the system as the boot drive, for installation.

    If you want to setup anything more complex as you may want to setup RAID or a specific partion layout, you need to use the curtin syntax.

    It is not immediately clear how to setup a RAID configuration based on the available documentation.

    Fortunately, the new installer supports creating a RAID configuration or custom partition layout if you perform a manual install.

    It turns out that when the manual installation is done you can find the cloud-init user-data YAML for this particular configuration in the following file:

        /var/log/installer/autoinstall-user-data
    

    I think this is extremely convenient.

    So to build a proper RAID1-based installation, I followed these instructions.

    So what does the YAML for this RAID setup look like?

    This is the storage section of my user-data file (brace yourself):

    storage:
        config:
        - {ptable: gpt, serial: VBOX_HARDDISK_VB50546281-4e4a6c24, path: /dev/sda, preserve: false,
          name: '', grub_device: true, type: disk, id: disk-sda}
        - {ptable: gpt, serial: VBOX_HARDDISK_VB84e5a275-89a2a956, path: /dev/sdb, preserve: false,
          name: '', grub_device: true, type: disk, id: disk-sdb}
        - {device: disk-sda, size: 1048576, flag: bios_grub, number: 1, preserve: false,
          grub_device: false, type: partition, id: partition-0}
        - {device: disk-sdb, size: 1048576, flag: bios_grub, number: 1, preserve: false,
          grub_device: false, type: partition, id: partition-1}
        - {device: disk-sda, size: 524288000, wipe: superblock, flag: '', number: 2, preserve: false,
          grub_device: false, type: partition, id: partition-2}
        - {device: disk-sdb, size: 524288000, wipe: superblock, flag: '', number: 2, preserve: false,
          grub_device: false, type: partition, id: partition-3}
        - {device: disk-sda, size: 1073741824, wipe: superblock, flag: '', number: 3,
          preserve: false, grub_device: false, type: partition, id: partition-4}
        - {device: disk-sdb, size: 1073741824, wipe: superblock, flag: '', number: 3,
          preserve: false, grub_device: false, type: partition, id: partition-5}
        - {device: disk-sda, size: 9136242688, wipe: superblock, flag: '', number: 4,
          preserve: false, grub_device: false, type: partition, id: partition-6}
        - {device: disk-sdb, size: 9136242688, wipe: superblock, flag: '', number: 4,
          preserve: false, grub_device: false, type: partition, id: partition-7}
        - name: md0
          raidlevel: raid1
          devices: [partition-2, partition-3]
          spare_devices: []
          preserve: false
          type: raid
          id: raid-0
        - name: md1
          raidlevel: raid1
          devices: [partition-4, partition-5]
          spare_devices: []
          preserve: false
          type: raid
          id: raid-1
        - name: md2
          raidlevel: raid1
          devices: [partition-6, partition-7]
          spare_devices: []
          preserve: false
          type: raid
          id: raid-2
        - {fstype: ext4, volume: raid-0, preserve: false, type: format, id: format-0}
        - {fstype: swap, volume: raid-1, preserve: false, type: format, id: format-1}
        - {device: format-1, path: '', type: mount, id: mount-1}
        - {fstype: ext4, volume: raid-2, preserve: false, type: format, id: format-2}
        - {device: format-2, path: /, type: mount, id: mount-2}
        - {device: format-0, path: /boot, type: mount, id: mount-0}
    

    That's quite a long list of instructions to 'just' setup a RAID1. Although I understand all the steps involved, I'd never have come up with this by myself on short notice using just the documentation, so I think that the autoinstall-user-data file is a life saver.

    After the manual installation, creating a RAID1 mirror, I copied the configuration above into my own custom user-data YAML. Then I performed an unattended installation and it worked on the first try.

    So if you want to add LVM into the mix or make some other complex storage configuration, the easiest way to automate it, is to first do a manual install and then copy the relevant storage section from the autoinstall-user-data file to your custom user-data file.

    Example user-data file for download

    I've published a working user-data file that creates a RAID1 here.

    You can try it out by creating a virtual machine with two (virtual) hard drives. I'm assuming that you have a PXE boot environment setup.

    Obviously, you'll have to change the network settings for it to work.

    Generating user passwors

    If you do want to logon onto the console with the default user, you must generate a salt+password hash and copy/past that into the user-data file.

    I juse the 'mkpasswd' command for this like so:

    mkpasswd -m sha-512
    

    The mkpasswd utility is part of the 'whois' package.

    Closing words

    For those who are still on Ubuntu Server 18.04 LTS, there is no need for immediate action as this version is supported until 2023. Only support for new hardware has come to an end for the 18.04 LTS release.

    At some point, some time and effort will be required to migrate towards the new Autoinstall solution. Maybe this blogpost helps you with this transition.

    It took me a few evenings to master the new user-data solution, but the fact that a manual installation basically results in a perfect pre-baked user-data file3 is a tremendous help.

    I think I just miss the point of all this effort of revamping installers but maybe I'm not hurt by the limitations of the older existing solution. If you have any thoughts on this, feel free to let me know in the comments.


    1. I had to discover this information on StackExchange somewhere down below in this very good article after experiencing problems. 

    2. I tried 384 MB and it didn't finish, just got stuck. 

    3. /var/log/installer/autoinstall-user-data 

    Tagged as : Linux
  2. ZFS: Performance and Capacity Impact of Ashift=9 on 4K Sector Drives

    Thu 31 July 2014

    Update 2014-8-23: I was testing with ashift for my new NAS. The ashift=9 write performance deteriorated from 1.1 GB/s to 830 MB/s with just 16 TB of data on the pool. Also I noticed that resilvering was very slow. This is why I decided to abandon my 24 drive RAIDZ3 configuration.

    I'm aware that drives are faster at the outside of the platter and slower on the inside, but the performance deteriorated so dramatically that I did not wanted to continue further.

    My final setup will be a RAIDZ2 18 drive VDEV + RAIDZ2 6 drive VDEV which will give me 'only' 71 TiB of storage, but read performance is 2.6 GB/s and write performance is excellent at 1.9 GB/s. I've written about 40+ TiB to the array and after those 40 TiB, write performance was about 1.7 GB/s, so still very good and what I would expect as drives fill up.

    So actually, based on these results, I have learned not to deviate from the ZFS best practices too much. Use ashift=12 and put drives in VDEVS that adhere to the 2^n+parity rule.

    The uneven VDEVs (18 disk vs. 6 disks) are not according to best practice but ZFS is smart: it distributes data across the VDEVs based on their size. So they fill up equally.


    Choosing between ashift=9 and ashift=12 for 4K sector drives is not always a clear cut case. You have to choose between raw performance or storage capacity.

    My testplatform is Debian Wheezy with ZFS on Linux. I'm using a system with 24 x 4 TB drives in a RAIDZ3. The drives have a native sector size of 4K, and the array is formatted with ashift=12.

    First we create the array like this:

    zpool create storage -o ashift=12 raidz3 /dev/sd[abcdefghijklmnopqrstuvwx]
    

    Note: NEVER use /dev/sd? drive names for an array, this is just for testing, always use /dev/disk/by-id/ names.

    Then we run a simple sequential transfer benchmark with dd:

    root@nano:/storage# dd if=/dev/zero of=ashift12.bin bs=1M count=100000 
    100000+0 records in
    100000+0 records out
    104857600000 bytes (105 GB) copied, 66.4922 s, 1.6 GB/s
    root@nano:/storage# dd if=ashift12.bin of=/dev/null bs=1M
    100000+0 records in
    100000+0 records out
    104857600000 bytes (105 GB) copied, 42.0371 s, 2.5 GB/s
    

    This is quite impressive. With these speeds, you can saturate 10Gbe ethernet. But how much storage space do we get?

    df -h:

    Filesystem                            Size  Used Avail Use% Mounted on
    storage                                69T  512K   69T   1% /storage
    

    zfs list:

    NAME      USED  AVAIL  REFER  MOUNTPOINT
    storage  1.66M  68.4T   435K  /storage
    

    Only 68.4 TiB of storage? That's not good. There should be 24 drives minus 3 for parity is 21 x 3.6 TiB = 75 TiB of storage.

    So the performance is great, but somehow, we lost about 6 TiB of storage, more than a whole drive.

    So what happens if you create the same array with ashift=9?

    zpool create storage -o ashift=9 raidz3 /dev/sd[abcdefghijklmnopqrstuvwx]
    

    These are the benchmarks:

    root@nano:/storage# dd if=/dev/zero of=ashift9.bin bs=1M count=100000 
    100000+0 records in
    100000+0 records out
    104857600000 bytes (105 GB) copied, 97.4231 s, 1.1 GB/s
    root@nano:/storage# dd if=ashift9.bin of=/dev/null bs=1M
    100000+0 records in
    100000+0 records out
    104857600000 bytes (105 GB) copied, 42.3805 s, 2.5 GB/s
    

    So we lose about a third of our write performance, but the read performance is not affected, probably by read-ahead caching but I'm not sure.

    With ashift=9, we do lose some write performance, but we can still saturate 10Gbe.

    Now look what happens to the available storage capacity:

    df -h:

    Filesystem                         Size  Used Avail Use% Mounted on
    storage                             74T   98G   74T   1% /storage
    

    zfs list:

    NAME      USED  AVAIL  REFER  MOUNTPOINT
    storage   271K  73.9T  89.8K  /storage
    

    Now we have a capacity of 74 TiB, so we just gained 5 TiB with ashift=9 over ashift=12, at the cost of some write performance.

    So if you really care about sequential write performance, ashift=12 is the better option. If storage capacity is more important, ashift=9 seems to be the best solution for 4K drives.

    The performance of ashift=9 on 4K drives is always described as 'horrible' but I think it's best to run your own benchmarks and decide for yourself.

    Caveat: I'm quite sure about the benchmark performance. I'm not 100% sure how reliable the reported free space is according to df -h or zfs list.

    Edit: I have added a bit of my own opinion on the results.

    Tagged as : ZFS Linux
  3. Linux: Script That Creates Table of Network Interface Properties

    Thu 15 August 2013

    My server has 5 network interfaces and I wanted a quick overview of some properties. There may be an existing linux command for this but I couldn't find it so I quickly wrote my own script (download).

    This is the output:

    showinterfacesimage

    The only requirement for this script is that you have 'ethtool' installed.

    Update 2013-08-17

    I recreated the script in python (download) so I can just dynamically format the table and not use ugly hacks I used in the bash script.

    Tagged as : Linux Networking

Page 1 / 5