DNS security: encryption and abuse

Once you know how DNS works and what rides on the wire, two security questions follow. DNS is cleartext, so who gets to read it? And it is trusted and rarely blocked, so who abuses that? This post covers both: encrypting the lookup, then how malware turns the same channel against you.

Encrypting DNS

Classic DNS (Do53) is cleartext on port 53: your ISP reads, and can rewrite, every name you look up. Three encrypted transports fix that, to different degrees. For the lookup this protects, start with how DNS works.

Switch between them to see what still leaks:

you resolver

Do53 Classic DNS

Q: hugosibony.com A · A: 104.21.86.12

port UDP / TCP · 53 RFC 1035

Encapsulation

  • DNS message question + answer, plaintext
  • UDP :53 no encryption layer at all

= encrypted, unreadable on the wire

What your ISP still sees

Everything. The exact name you asked for and the IP you got back, in cleartext. Your ISP can log it, sell it, or rewrite the answer.

DoT DNS over TLS

encrypted TLS record

port TCP · 853 RFC 7858

Encapsulation

  • DNS message the query bytes
  • TLS encrypts everything above
  • TCP :853 dedicated DNS-over-TLS port

= encrypted, unreadable on the wire

What your ISP still sees

That you sent DNS to this resolver, and nothing about the names. Port 853 is a dead giveaway that it is DNS, so it can be blocked wholesale.

DoQ DNS over QUIC

encrypted QUIC packet

port UDP · 853 RFC 9250

Encapsulation

  • DNS message the query bytes
  • QUIC (TLS 1.3) encrypts everything above
  • UDP :853 QUIC transport, not TCP

= encrypted, unreadable on the wire

What your ISP still sees

Same as DoT: which resolver you chose, and that it is DNS (the UDP/853 signature). Query names stay hidden. Lower latency than DoT.

DoH DNS over HTTPS

encrypted HTTPS request

port TCP · 443 RFC 8484

Encapsulation

  • DNS message POST /dns-query · application/dns-message
  • HTTP/2 wraps the message as an HTTPS request
  • TLS encrypts everything above
  • TCP :443 same port as all HTTPS

= encrypted, unreadable on the wire

What your ISP still sees

Only that you opened an HTTPS connection to this IP, indistinguishable from loading any website. Hardest to block or profile.

In every encrypted mode the observer still learns the resolver's IP, and the TLS SNI can leak which resolver you picked, unless Encrypted Client Hello (ECH) hides it. Your individual queries stay private either way.

The catch in every encrypted mode: the observer still sees the resolver’s IP, and the TLS SNI can reveal which resolver you picked unless Encrypted Client Hello (ECH) hides it. Your individual queries stay private either way.

To turn encryption on and prove it works, see which resolver to choose.

How malware abuses DNS

DNS is trusted, everywhere, and rarely blocked, which is why attackers use it: tunnelling data out, running command-and-control, generating throwaway domains to dodge blocklists. This is the defensive view: how each technique works, real cases, and how it’s caught. For the protocol itself, start with how DNS works.

The attacker owns a domain and runs its authoritative server. Data is encoded into the subdomain labels going out, and answers ride back in TXT / NULL / CNAME records. The victim only ever talks to its normal resolver, which forwards the queries, so the tunnel slips straight through egress filtering.

n0uqbn3xkr2.af9d21.tunnel.example. IN TXT?

base32 payload · label ≤ 63 B, name ≤ 255 B · low throughput, high latency

Documented in the wild

How it's caught: Long, high-entropy names; many unique subdomains under one parent domain; a burst of TXT / NULL queries from one host.

Commands and results are carried in DNS fields, often as low-and-slow beacons that blend into ordinary lookups. DNS is rarely blocked on corporate networks, so it is a reliable back channel.

Documented in the wild

  • DNSMessenger fileless PowerShell RAT with a two-way channel over TXT records (Cisco Talos, 2017)
  • OilRig / Helminth, BONDUPDATER Iranian-nexus trojans using DNS tunneling for C2, TXT-based (Unit 42)
  • SUNBURST encoded victim identity into avsvmcloud[.]com subdomains, then handed off via CNAME (Mandiant)

How it's caught: Regular, low-volume queries to a single domain (beaconing); disproportionate TXT traffic from one endpoint.

Malware algorithmically generates thousands of candidate domains from a seed plus the current date, then resolves them until it hits the one the operator registered for that day. You cannot pre-block domains you cannot predict, which defeats static blocklists and takedowns.

kqkz0rmxvopht.info NXDOMAIN tj3n8wq2ldacs.info NXDOMAIN ptvbuk9raewf.info 45.13.x.x (live C2)

the client walks its daily list until one answers

Documented in the wild

  • Conficker UTC-date DGA; variants A/B made ~250 domains/day, variant C escalated to ~50,000/day
  • SUNBURST DGA-style subdomain scheme encoding the victim, under one fixed parent domain (Mandiant)

How it's caught: Bursts of NXDOMAIN answers for random-looking names; newly registered domains; ML DGA classifiers over query logs.

Exfiltration

ATT&CK T1048

Stolen data is chunked, encoded, and packed into the query names sent to the attacker's authoritative server, which reassembles it from its query logs. Mechanically it is tunneling, but usually outbound-only and aimed at bulk theft.

4d616c776172655f63617264.exfil.example. IN A?

hex/base32 chunk of the stolen record in each query

Documented in the wild

  • FrameworkPOS point-of-sale malware that exfiltrated stolen credit-card data over DNS (MITRE T1048)

How it's caught: High volume of unique, long subdomains to one domain; query names that carry data-shaped entropy.

One domain is mapped to a large, churning pool of compromised proxy hosts by rotating its records with very short TTLs. Single flux cycles the A records; double flux also cycles the NS records, hiding the true origin behind a second layer.

bad.example. 30 IN A 91.x.x.1 bad.example. 30 IN A 203.x.x.7 (same name, new IPs every few minutes)

TTL ~30–300 s so blocklists never catch up

Documented in the wild

  • Fast Flux DNS used broadly by botnets and phishing kits to keep C2 resilient against IP blocking (MITRE T1568.001)

How it's caught: A single name resolving to many IPs with very low TTLs; rapidly changing A / NS answers.

DoH abuse

The same encryption that protects you also hides malware. By doing DNS over HTTPS to a public resolver, an implant conceals both its queries and the fact that DNS is happening from monitors that only watch UDP/53.

Documented in the wild

  • Godlua widely cited as the first malware to use DoH, pulled C2 addresses via TXT records over DoH (360 Netlab, 2019)

How it's caught: Endpoints reaching public DoH resolvers instead of the enterprise one; force all DNS through an inspected internal resolver.

Defenders lean on passive DNS (retro-hunting), Response Policy Zones (a resolver-level DNS firewall), newly-registered-domain and threat-intel feeds, and egress control, only the enterprise resolver may reach port 53.

References

Encryption

  • RFC 7858: DNS over TLS (DoT), TCP 853.
  • RFC 8484: DNS over HTTPS (DoH), application/dns-message over TCP 443.
  • RFC 9250: DNS over QUIC (DoQ), UDP 853.
  • RFC 1035: classic DNS (Do53) over UDP/TCP 53.
  • RFC 9460: the HTTPS record and its Encrypted Client Hello (ECH) hint.

Abuse (MITRE ATT&CK)

  • T1071.004: Application Layer Protocol: DNS (tunnelling and C2).
  • T1568.002: Domain Generation Algorithms.
  • T1568.001: Fast Flux DNS.
  • T1048: Exfiltration Over Alternative Protocol.

Campaign and technique reporting

DNS series

  1. How DNS works
  2. Inside a DNS packet
  3. DNS security: encryption and abuse (this post)

Practical companion: Which resolver to choose.

esc