How DNS Traffic Can Be a Canary in the Coal Mine

While it's noisy and difficult to monitor, I love looking at DNS traffic. It's a gift that keeps on giving. I like to think of it as a canary in your network's coal mine.

DNS is the foundation of your network. If DNS breaks, it takes everything with it. DNS is also really noisy. The sheer volume of DNS queries makes it difficult to monitor and secure.

DNS usually goes over user datagram protocol or UDP, which speeds up transmissions by allowing data to transfer before the receiving party has an agreement in place. Basically, UDP sends traffic and hopes it gets there. It's also spoofable and vulnerable to attacks.

Most security professionals depend on logs to identify suspicious activity, such as unusual queries. Relying on those logs as your first line of defense comes with risks, especially if they're incomplete. Because DNS traffic is so noisy, it's a challenge to log.

Recently, I investigated an incident with a government organization where some findings indicated that the organization was a victim of a DNS attack.

Investigating DNS Attacks

You can detect amplification attacks (and, by the way, DNS tunneling) with either payload analysis or traffic analysis. Payload analysis looks at the contents of DNS requests and responses. Factors such as unusual host names or differences between the size of the request versus the response can help identify suspicious activity.

For example, in the SolarWinds SUNBURST attack, attackers evaded detection by taking advantage of known weaknesses within DNS to hide command-and-control traffic. An estimated 18,000 companies were impacted by the SUNBURST DNS tactics, putting organizations on high alert for similar activity.

When we investigated the incident at the government agency, we were surprised to see the amount of DNS "ANY" requests—the DNS equivalent of "give me all your things." Attackers use this query to get all the information about a domain, and armed with that, they can take an organization offline.

Just think—if a person (let's say Alice) forges a request to Bob saying it's "from Charlie," she can send a 75-character request. Bob will turn around and send back 482 characters to Charlie, and that reply can inadvertently bring down Charlie's servers. Alice laughs at Charlie. If Alice has 10,000 friends (or 10,000 bots) all doing the same thing—uh oh, sucks to be Charlie. This is called a DNS amplification attack.

A Canary in the Coal Mine

Usually, clients pick a random port, called the ephemeral port, each time they make a DNS request. Seeing the same port for thousands of requests should set off alarm bells. That's exactly what we saw: thousands and thousands of DNS requests coming from the same client port from a single address on the internet, to a single name server hosted by this ExtraHop customer.

To add to the bizarreness, that server wasn't supposed to be in production. Using ExtraHop Reveal(x) to view the activity at the packet level, we saw the same query made more than 40,000 times in an hour using the same "ANY" method from one external IP and the same client port.

Tools exist that can review large swathes of the internet in minutes, and just because your infrastructure isn't in production doesn't mean the internet can't find you. Based on the DNS requests, it appeared that someone found this server and was trying to use it.

The team checked, but they weren't logging DNS requests. Without Reveal(x), this activity had the potential to go undetected.

We have a few theories about what was happening based on some of the details. Using built-in ARIN 'whois' lookup in Reveal(x), we determined the client IP address was owned by a cable provider in the midwest. It was most likely a home internet connection, leaving me with this wild guess:

• Alice and Bob were playing Call of Warcraft: New Vegas (or whatever).
• Bob took out Alice.
• Alice is nonplussed.
• Alice decides to take Bob offline.
• Alice uses a click-n-crash tool (or a for-pay denial of service provider).
• Alice forges DNS requests that appear to come from Bob.
• DNS requests hit Charlie's (the ExtraHop customer's) DNS infrastructure.
• DNS responses go back to Bob.
• Multiply this by 1,000 or 5,000 different endpoints.
• Bob is offline.
• Alice is pleased.
• Achievement unlocked!

If this was a click-n-crash attempt, our work that day put a kink in someone's plans. In the end, this government organization was able to close critical visibility gaps, ultimately leading the network team to isolate the server to bring the organization's internet usage back to normal.