Louwrentius

Purpose of this blogpost

I'm documenting how I setup my PXE server for myself and maybe somebody else may also find it useful. It's mostly focussed on diminishing the role of the TFTP server as much as possible as TFTP is very slow.

My goal is to PXE HTTP boot Debian installations on both virtual machines or physical machines.

It should also be possible to fully automatically provision a machine through PXE targeted to a specific machine.

I'm also moving away from BIOS boot and try to configure all hardware using UEFI.

Why TFTP is a problem

TFTP is extremely slow. I've never been able to get an individual client to load a file over TFTP faster than maybe 3 MB/s.

So the trick is to use TFTP only to load the iPXE boot loader and use the HTTP boot support of iPXE to load all subsequent data / files, wich will achieve much higher download speeds.

Please note that modern UEFI-based server hardware does support HTTP boot natively and since I don't have such hardware, I can't cover that in this blogpost, but not all computers supporting UEFI, support UEFI HTTP boot and this blogpost is relevant for those systems.

PXE TFTP + HTTP Boot Sequence

To understand the entire boot process, let's take a look at the entire end-to-end process.

1. PXE DHCP: the client boots and requests an IP address and reads the TFTP server from the DHCP server response
2. PXE TFTP: the client gets the iPXE PXE boot loader over TFTP (ipxe.efi)
3. iPXE DHCP: the client boots the iPXE boot loader and gets a DHCP address
4. iPXE HTTP: the iPXE client loads its configuration from a HTTP server
5. IPXE HTTP: the iPXE client loads the Linux kernal for unattended install
6. Debian Linux reads the preseed file over HTTP for unattended install.

So to summarise: we use the native UEFI (non-HTTP) PXE client to only bootstrap the iPXE (HTTP) PXE client that can load the rest of the files over HTTP.

KEA DHCP server

Since the ISC dhcp server that came standard with Debian Linux is end-of-life, we have to switch over to the KEA DHCP server.

DHCP pool with next-server configuration

This is the KEA DHCP pool configuration, which allows static IP address reservations outside the defined scope of the configured DHCP IP pool.

In this example the TFTP server is the next-server 10.10.10.1 configured here.

"next-server": "10.10.10.1",
        "pools": [
          {
            "option-data": [
              {
                "name": "broadcast-address",
                "data": "10.10.10.255"
              }
            ],
            "pool": "10.10.10.150 - 10.10.10.199"
          }
        ],
        "reservations-in-subnet": true,
        "reservations-out-of-pool": true,
        "reservations": [
          {
            "hostname": "example-server",
            "hw-address": "de:ad:be:ef:de:ad",
            "ip-address": "10.10.10.40",
            "boot-file-name": "pxelinux.0",
            "next-server": "10.10.10.1"
          },
          {
            "hostname": "kvm01",
            "hw-address": "aa:bb:cc3:dd:ee:ff",
            "ip-address": "10.10.10.51"
          },

KEA TFTP server configuration

These sections are there to push BIOS, UEFI and iPXE clients to different boot files.

"client-classes": [
      {
        "name": "XClient_iPXE",
        "test": "substring(option[77].hex,0,4) == 'iPXE'",
        "boot-file-name": "http://10.10.10.1/init.ipxe"
      },
      {
        "name": "UEFI-64-1",
        "test": "substring(option[60].hex,0,20) == 'PXEClient:Arch:00007'",
        "boot-file-name": "ipxe.efi"
      },
      {
        "name": "BIOS-64-1",
        "test": "substring(option[60].hex,0,20) == 'PXEClient:Arch:00000'",
        "boot-file-name": "ipxe.pxe"
      }

Notice how we can specify a boot-file-name pointing to a HTTP-based resource for the iPXE boot client.

TFTP setup

TFTP configuration

The file /etc/default/tftpd-hpa contains:

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure -R 30000:30100 --blocksize 1468"

iPXE setup

iPXE download software

Download the Network boot server files from this location

Example URL: https://github.com/ipxe/ipxe/releases/download/v2.0.0/ipxeboot.tar.gz

Extract the ipxe.efi file from the archive and put it in /srv/tftp

Understanding the iPXE boot script process

• I want to provide a basic boot menu where I can select the operating system and configuration I want to install.

• For some hosts, don't want to select the OS manually, but automatically install an OS when PXE boot is initiated.

So the process looks like this:

1. Load the init.ipxe boot script
2. The init.ipxe script loads the host-specific ipxe boot script if it exists for unattended OS install
3. Otherwise the manual OS menu is booted

iPXE initial boot script

This first ipxe script is loaded and checks if a boot script for this particular host exists in the boot directory. The name of the host is defined by DHCP in the static DHCP entry where the hostname is set.

#!ipxe

set boot-dir boot/
isset ${hostname} && chain --replace --autofree ${boot-dir}hostname-${hostname}.ipxe ||
chain --autofree uefi-boot-menu.ipxe

iPXE boot script for individual host

A host-specific boot script can look like this:

#!ipxe

set pxe-server http://10.10.10.1
set base-url ${pxe-server}

set debian-installer ${base-url}/debian-installer/amd64
kernel ${debian-installer}/linux
initrd ${debian-installer}/initrd.gz
imgargs linux auto=true priority=critical interface=eno1 url=http://10.10.10.1/preseed/preseed-debian-uefi-kvm.cfg elevator=noop -- quiet
boot

Notice how the kernel arguments specify a Debian preseed file.

Main iPXE manual boot menu

This iPXE script uefi-boot-menu.ipxe creates a simple syslinux-style boot menu where you can manually select which OS configuration to install.

This file is loaded if no host-specific boot script is found.

#!ipxe

# Basic setup
set menu-timeout 0
set submenu-timeout ${menu-timeout}
isset ${menu-default} || set menu-default Debian Installer

set pxe-server http://10.10.10.1
set base-url ${pxe-server}

###################### MAIN MENU ####################################
:start
menu iPXE Boot Menu 
item --gap --             ------------------------- Operating Systems ----------------------------
item --key d menu-debian Debian Installer
item --gap --             ------------------------- PXE options -------------------------------
item --key c config       Configure settings
item shell                Drop to iPXE shell
item reboot               Reboot computer
item --key x exit         Exit iPXE and continue BIOS boot
choose --timeout ${menu-timeout} --default ${menu-default} selected || goto cancel
set menu-timeout 0
goto ${selected}

:cancel
echo You cancelled the menu, dropping you to a shell

:shell
echo Type 'exit' to get the back to the menu
shell
set menu-timeout 0
set submenu-timeout 0
goto start

:failed
echo Booting failed, dropping to shell
goto shell

:reboot
reboot

:exit
exit

:config
config
goto start

:back
set submenu-timeout 0
clear submenu-default
goto start


###################### Debian MENU ################################

:menu-debian
menu Debian Installer
item debian-manual Debian 13 (Trixie) Manual install 
item debian-unattended-tmm Debian 13 (Trixie) Unattended install UEFI (ELITE DESK)
item debian-unattended-tmm-kvm Debian 13 (Trixie) Unattended install UEFI KVM SERVER (ELITE DESK)
item debian-unattended-bios Debian 13 (Trixie) Unattended install BIOS
item --key 0x08 back Back to top menu...
choose selected && goto ${selected} || goto start

:debian-manual
set debian-installer ${base-url}/debian-installer/amd64
kernel ${debian-installer}/linux
initrd ${debian-installer}/initrd.gz
boot || goto failed
goto start

:debian-unattended
set debian-installer ${base-url}/debian-installer/amd64
kernel ${debian-installer}/linux
initrd ${debian-installer}/initrd.gz
imgargs linux auto=true priority=critical interface=eno1 url=http://10.10.10.1/preseed/preseed-debian-uefi.cfg elevator=noop -- quiet
boot || goto failed
goto start

:debian-unattended-bios
set debian-installer ${base-url}/debian-installer/amd64
kernel ${debian-installer}/linux
initrd ${debian-installer}/initrd.gz
imgargs linux auto=true priority=critical interface=eno1 url=http://10.10.10.1/preseed/preseed-debian.cfg elevator=noop -- quiet
boot || goto failed
goto start

The Domain Name System exists because it's difficult for people to remember IP addresses (185.15.59.224) and much easier to remember domain names (wikipedia.org).

Regarding internet-accessible services, it makes sense to publish websites, API endpoints or similar services using DNS, as people have to interfact with them. The added benefit of a domain name is that the associated IP address can change without the client being affected.

This article isn't against DNS for public services, but it questions if we should use DNS for internal IT infrastructure (independent of cloud vs. onprem)

It's always DNS

Although DNS can be a very beneficial service, it can also become a liability. If you want a reliable system, you want as little components as possible. Every additional component adds a potential risk of failure. In addition, more components may create unforeseen behaviour and interactions that can cause outages (circular dependancies, and so on). If you can avoid adding components, you'll have a better chance of building a reliable system.

Within the IT operations space, DNS has made a bit of a name for itself. Many may remember this little haiku.

It’s not DNS
There’s no way it’s DNS
It was DNS

(source)

There are multiple(1) high-profile(2) incidents where DNS was involved. In these linked cases, the root-cause of the incident isn't the DNS system itself. Yet, because the root-cause affects the DNS service - which is in the critical path for virtually all services - the incident has such a huge impact.

The Facebook / Meta outage was so significant because it locked people out of buildings (physical access) due to 'circular' dependancies on DNS being available. Again, it can be said that the circular dependancy is the root-cause, but the blast radius of DNS is in many cases so enormous that it may be difficult to have a clear end-to-end picture of potential risk.

The case against DNS for internal IT infrastructure

From the perspective of IT operations, DNS has a drawback: DNS clients cache DNS records based on TTL. Different DNS client implementations can behave differently, but even if you have a fairly homogenous environment, the only way to assure clients (in this case other servers) use the updated IP address, is to control them and force a DNS refresh.

That got me thinking, why would we use DNS for infrastructure services? It isn't necessary for machine-to-machine communication. Instead of configuring domain names that may not resolve, we can just directly inject the appropriate IP address(ess) into configuration files. It's easy to configure systems with tools like Ansible or pyinfra at scale.

The counter argument could be that DevOPS / platform engineers are also humans, and it's much easier to spot misconfigurations or to troubleshoot if domain names are configured Instead of IP addresses.

Fortunately, we still have /etc/hosts, which we can easily provision. Still no DNS service required! This way, we can configure domain names and pretend to use DNS. I also suspect that DNS queries against /etc/hosts are quite responsive.

DNS as generic security risk

As of today, most network traffic is encrypted by default, or tunneled through an encrypted channel. DNS is - by default - the exception. Regarding internal IT infrastructure (cloud or 'onprem'), the network may be considered as a secure environment. An attack on the DNS service, spoofing packets, and so on, can be very disruptive though. Setting up DNSSEC may alleviate this problem, but that also introduces another administrative burden with it's own risk of misconfiguration. It's yet another layer of complexity. And we assume that internal infrastructure supports DNSSEC.

DNS as an Egress Exfiltration risk

Because egress filtering (filtering of outbound connections) can be cumbersome, it's often omitted, because the systems involved are 'trusted'. This is unfortunate as this makes life easier for an attacker. Any kind of resource required for an attack can be acquired on the vulnerable system with a simple outbound query towards the internet. Proper egress filtering of network traffic can be the difference between a succesfull and unsuccessful hacking attempt.

A lack of egress filtering also makes it much easier for an attacker to exfiltrate data. And the thing is: any IP protocol can be used to exfiltrate data, including DNS¹.

This is how: the attacker gets a domain runs their internet-accessible authoritative nameserver for this domain. Now the attacker can make DNS requests to said domain like sensitivedata.evil.domain from the hacked system and you can extract all the data from the rogue DNS server logs².

Although a hacked server may not be able to directly interact with the attacker-controlled DNS server, by issuing DNS requests for the attacker-controlled domain, these requests will pass the local forwarding DNS server and be forwarded towards the attacker-controlled authoritative DNS server. See also tools like dnscat2 or iodine

Due to this risk, there is a case to be made, to - at least - not allow systems to query public DNS records. As servers may need to interfact with services on the internet (update servers, APIs, and so on), such access can be facilitated by a proxy server using allow-listed domains.

Evaluation and closing words

In the end, everything is a tradeoff, where people must balance benefits and drawbacks against the context of their infrastructure, their particular risk appetite and even organisational structure and culture.

That said, I think it's reasonable to explore if DNS can be avoided altogether within the IT infrastructure to increase reliability and robustness.

Feel free to share your thoughts and feelings about this if you feel so inclined. Maybe leave a comment on the Hacker News thread.

Postscript on HN response (update)

Update: I do find the responses on the HN thread very interesting from an antropological perspective. How people respond and what implicit assumptions they make. One thing that I found interesting is how many people implicitly assume an environment is 'kubernetes-like', and they focus on 'service discovery'. Problems do arise if you have many IP address mutations for your infrastructure, especially at scale. Because I was thinking about very old-fashioned virtual machine or bare-metal style environment at a small to medium scale (a scale that is much more common I believe, most environments aren't Facebook/Google or even 1/100000 that size).

I realised that I should have given more context in this article, under which conditions I think my 'DNS alternative' could work well and acknowledge where it would not. Anyway, acknowledging my fault for not creating enough context, I do find the knee-jerk responses, hostility and sometimes personal attacks curious. Only very few* people seem to pause, stay calm, think about my article, and acknowledge that it could work, within specific circumstances. Also many skipped a core part of the article which was thinking about risk and complexity, in some sense it wasn't only about DNS.

I also just realise that I made a very important mistake. Servers hosting applications don't need DNS resolving for the internet. I got a feeling from some responses that I was proposing that /etc/hosts would be updated with thousands of DNS records for external internet-facing services. I was only talking about internal infrastructure where internal services, like other servers, or NTP, SMTP, or whatever is configured. I think I did address this assumption in the article, but not nearly explicitly enough.