Louwrentius - Infrastructure

DNS is for people - not for IT infrastructure

2026-06-03T12:00:00+02:00

The Domain Name System exists because it's difficult for people to remember IP addresses (185.15.59.224) and much easier to remember domain names (wikipedia.org).

Regarding internet-accessible services, it makes sense to publish websites, API endpoints or similar services using DNS, as people have to interfact with them. The added benefit of a domain name is that the associated IP address can change without the client being affected.

This article isn't against DNS for public services, but it questions if we should use DNS for internal IT infrastructure (independent of cloud vs. onprem)

It's always DNS

Although DNS can be a very beneficial service, it can also become a liability. If you want a reliable system, you want as little components as possible. Every additional component adds a potential risk of failure. In addition, more components may create unforeseen behaviour and interactions that can cause outages (circular dependancies, and so on). If you can avoid adding components, you'll have a better chance of building a reliable system.

Within the IT operations space, DNS has made a bit of a name for itself. Many may remember this little haiku.

It’s not DNS
There’s no way it’s DNS
It was DNS

(source)

There are multiple(1) high-profile(2) incidents where DNS was involved. In these linked cases, the root-cause of the incident isn't the DNS system itself. Yet, because the root-cause affects the DNS service - which is in the critical path for virtually all services - the incident has such a huge impact.

The Facebook / Meta outage was so significant because it locked people out of buildings (physical access) due to 'circular' dependancies on DNS being available. Again, it can be said that the circular dependancy is the root-cause, but the blast radius of DNS is in many cases so enormous that it may be difficult to have a clear end-to-end picture of potential risk.

The case against DNS for internal IT infrastructure

From the perspective of IT operations, DNS has a drawback: DNS clients cache DNS records based on TTL. Different DNS client implementations can behave differently, but even if you have a fairly homogenous environment, the only way to assure clients (in this case other servers) use the updated IP address, is to control them and force a DNS refresh.

That got me thinking, why would we use DNS for infrastructure services? It isn't necessary for machine-to-machine communication. Instead of configuring domain names that may not resolve, we can just directly inject the appropriate IP address(ess) into configuration files. It's easy to configure systems with tools like Ansible or pyinfra at scale.

The counter argument could be that DevOPS / platform engineers are also humans, and it's much easier to spot misconfigurations or to troubleshoot if domain names are configured Instead of IP addresses.

Fortunately, we still have /etc/hosts, which we can easily provision. Still no DNS service required! This way, we can configure domain names and pretend to use DNS. I also suspect that DNS queries against /etc/hosts are quite responsive.

DNS as generic security risk

As of today, most network traffic is encrypted by default, or tunneled through an encrypted channel. DNS is - by default - the exception. Regarding internal IT infrastructure (cloud or 'onprem'), the network may be considered as a secure environment. An attack on the DNS service, spoofing packets, and so on, can be very disruptive though. Setting up DNSSEC may alleviate this problem, but that also introduces another administrative burden with it's own risk of misconfiguration. It's yet another layer of complexity. And we assume that internal infrastructure supports DNSSEC.

DNS as an Egress Exfiltration risk

Because egress filtering (filtering of outbound connections) can be cumbersome, it's often omitted, because the systems involved are 'trusted'. This is unfortunate as this makes life easier for an attacker. Any kind of resource required for an attack can be acquired on the vulnerable system with a simple outbound query towards the internet. Proper egress filtering of network traffic can be the difference between a succesfull and unsuccessful hacking attempt.

A lack of egress filtering also makes it much easier for an attacker to exfiltrate data. And the thing is: any IP protocol can be used to exfiltrate data, including DNS¹.

This is how: the attacker gets a domain runs their internet-accessible authoritative nameserver for this domain. Now the attacker can make DNS requests to said domain like sensitivedata.evil.domain from the hacked system and you can extract all the data from the rogue DNS server logs².

Although a hacked server may not be able to directly interact with the attacker-controlled DNS server, by issuing DNS requests for the attacker-controlled domain, these requests will pass the local forwarding DNS server and be forwarded towards the attacker-controlled authoritative DNS server. See also tools like dnscat2 or iodine

Due to this risk, there is a case to be made, to - at least - not allow systems to query public DNS records. As servers may need to interfact with services on the internet (update servers, APIs, and so on), such access can be facilitated by a proxy server using allow-listed domains.

Evaluation and closing words

In the end, everything is a tradeoff, where people must balance benefits and drawbacks against the context of their infrastructure, their particular risk appetite and even organisational structure and culture.

That said, I think it's reasonable to explore if DNS can be avoided altogether within the IT infrastructure to increase reliability and robustness.

Feel free to share your thoughts and feelings about this if you feel so inclined. Maybe leave a comment on the Hacker News thread.

Postscript on HN response (update)

Update: I do find the responses on the HN thread very interesting from an antropological perspective. How people respond and what implicit assumptions they make. One thing that I found interesting is how many people implicitly assume an environment is 'kubernetes-like', and they focus on 'service discovery'. Problems do arise if you have many IP address mutations for your infrastructure, especially at scale. Because I was thinking about very old-fashioned virtual machine or bare-metal style environment at a small to medium scale (a scale that is much more common I believe, most environments aren't Facebook/Google or even 1/100000 that size).

I realised that I should have given more context in this article, under which conditions I think my 'DNS alternative' could work well and acknowledge where it would not. Anyway, acknowledging my fault for not creating enough context, I do find the knee-jerk responses, hostility and sometimes personal attacks curious. Only very few* people seem to pause, stay calm, think about my article, and acknowledge that it could work, within specific circumstances. Also many skipped a core part of the article which was thinking about risk and complexity, in some sense it wasn't only about DNS.

I also just realise that I made a very important mistake. Servers hosting applications don't need DNS resolving for the internet. I got a feeling from some responses that I was proposing that /etc/hosts would be updated with thousands of DNS records for external internet-facing services. I was only talking about internal infrastructure where internal services, like other servers, or NTP, SMTP, or whatever is configured. I think I did address this assumption in the article, but not nearly explicitly enough.

Don't forget about services like NTP or ICMP. ↩
I have demonstrated this attack using this exact method with a domain I own for a customer that thought they had properly prevented egress traffic, including blocking NTP and ICMP. ↩

Improve privacy by running a DNS server without forwarder

2026-03-25T12:00:00+01:00

Disclaimer: no AI was used to write this article.

If you run a DNS server at home or for an organisation, it is likely that you configured a public DNS server as a forwarder¹.

This is not required!

Imagine we want to resolve the address 'wikipedia.org'.

If no forwarder is configured, a DNS server will behave like this:

Query the root server³ for an IP address of the .org top-level domain authoritative DNS server
Query the .org DNS server for an IP address of the wikipedia.org authoritative DNS server
Query the wikipedia.org DNS server for the IP address of 'wikipedia.org'

This is what the forwarder does when you query a domain that is not cached by said forwarder.

A forwarder responds faster?!

Yes, that's probably true, but I think it doesn't matter.

As a DNS forwarder caches DNS lookups, it will respond directly with the appropriate IP address. Only a single DNS query is required.

Without a forwarder, a domain lookup that is not cached, will require at least two or three DNS queries, the first time this domain is resolved. After this first set of queries, the domain is cached for the duration of the TTL.

I've performed a few manual lookups with 'dig' for domains that aren't cached, and most responses are <50ms. Depending on where you live and the quality of the network connection (in terms of latency) it may or may not be noticable, that's up to you to explore.

So what is the privacy angle?

You are exposing privacy-sensitive information to Google or other large companies.

If a DNS forwarder is configured, the forwarder will 'see' all your DNS queries and will be able to build a nice profile about you as a person or organisation.

It's up to you if you care about this 'risk'.

Running Pi-hole and BIND9 together

Pi-hole only supports forwarders?

Because Pi-hole requires a DNS forwarder to be configured, I'm also running BIND as the actual DNS server that performs the outbound DNS queries.

Both Pi-hole and BIND want to listen on UDP port 53, which won't work². In my case, I've configured BIND9 to listen on UDP port 5353 and configured Pi-hole to use 'BIND listen ip address#5353' as a forwarder.

This setup means that I benefit from the ad blocking by Pi-hole and won't send all my DNS requests to Google.

Help, my local DNS authoritative '.internal' zone isn't working anymore!

I'm running a local DNS domain 'xyz.internal' for servers and services in my network. I've also configured my DHCP server to tell clients the search domain is 'xyz.internal', so when I issue ssh server01 this will automatically resolve to server01.xyz.internal.

So to summarise, my BIND DNS server is both the DNS server that respons to client DNS lookups via Pi-hole for public domains, and it's also authoritative for the 'xyz.internal' domain.

However, there's a problem with this setup.

Pi-hole does not forward queries for the '.internal' domain because it considers upstream forwarding DNS servers as potentially hostile. Forwarding queries for '.internal' domains would expose potentially sensitive information.

This behaviour can be circumvented with the 'Conditional forwarding' option within Pi-hole. This allows DNS clients contacting the Pi-hole server to still query '.internal' domains. This is an example configuration line:

true,10.10.10.0/24,10.10.10.1#5353,xyz.internal

In this example, Pi-hole will forward DNS queries for the xyz.internal domain towards 10.10.10.1#5353 and everything will work as expected.

Q & A

How does the DNS server know how to contact the .org root server?

The DNS server is accompanied by a 'root zone file' containing the IP addresses for the root servers. As these IP addresses are not static, they need to be updated periodically. In Debian Linux (for example), this file is automatically updated through APT with the 'dns-root-data' package.

Probably an address from Google, Cloudflare or OpenDNS. ↩
It's possible to make Pi-hole and BIND listen on different interfaces, thus allowing both of them to use port 53 but that's another story and may not be that handy. ↩
A root server is not a single physical server but a group of physical servers, but I consider this an implementation detail. ↩

Sharing group_var variables across Ansible inventories

2025-12-20T12:00:00+01:00

The problem

Some time ago I worked with a customer that used Ansible to deploy a multitude of applications across their individual TAP¹ environments.

Unfortunately the Ansible inventory was setup in such a way that group_vars variables for a specific application were manually duplicated across each TAP environment.

Example of inventory folder hierarchy

Changing or adding a group_vars variable meant updating the group_vars file for each TAP environment. Making changes to group_vars in this environment is a cognitive burden: there is a high risk that one of the environments is missed by mistake.

The different TAP environments share the vast majority of all group_var variables, except for just a few variables that are environment-specific. There should be no reason to duplicate group_vars variables for each environment.

The solution

A shared folder symlinked into group_vars

Within the application inventory folder, We first create a folder called 000_shared that will contain all group_vars YAML files with variables that are shared across all TAP environments.

Next, we symlink this folder within each group_vars folder of each environment. Each environment also contains environment-specific configuration in one or more separate files.

With this example, any change within webserver.yaml will be applied to all TAP environments of the 'Alpha' application.

Sharing group_vars variables across applications

We can apply this trick one level higher: we can create a shared folder called 000_shared_global containing settings that are shared across all application environments.

In this example, we could change the DNS configuration for all environments by changing the settings in one, single file. But the global shared group_vars can be used to assure a (security) baseline between unrelated applications that do share similar technology (All Python or PHP applications).

Evaluation

Implementing this solution in an existing Ansible 'code base' can be a significant undertaking, but the long-term benefits seem worth it.

It reduces the amount of work required for changes
It reduces the risk of human error
It reduces the cognitive load

The good news is that you can migrate to this new style of inventory at your own pace, there is no need for a big-bang deployment. Changes can be made to one environment at a time.

Q & A

Why the leading 000 zeros in shared folder names?

Ansible parses group_vars YAML files according to their order within a directory. This means that the shared folders should be named with leading zeros, to assure that they are always parsed first.

This allows you to override shared group_vars settings for a particular environment, because these environment-specific files will be parsed later and their variable settings will be applied.

Just use one inventory for DTAP!

You can create a single inventory for test, acceptance and production, and separate the environments with groups. A shared group contains shared settings and environment-specific settings are put in the group_vars of the specific group. Unfortunately there is a risk:

Ansible documentation:

If you need to manage multiple environments, consider defining only the hosts of a single environment in each inventory. This way, it is harder to, for example, accidentally change the state of nodes inside the “test” environment when you wanted to update some “staging” servers.

Therefore, I would recommend against this approach.

Changes impact production?

Changing shared group_vars variables means that all environments are changed, including production. This requires proper code review and deployment processes that assures that said changes are only deployed to production when test and acceptance environments have been changed first.

Development, Test, Acceptance and Production ↩
When an inventory contains test, acceptance and prod environments it's easy to target all three environments by mistake. ↩