Uncovering Google DNS Activity with ExtraHop Reveal(x) Advisor Threat Hunting

The Observation

ExtraHop analysts observed a machine in a customer's network that was using Google for DNS. In production, you typically do not have servers reach directly to Google for DNS. With a click in ExtraHop Reveal(x), we immediately found a server in the customer's environment doing DNS lookups to a very suspicious IP address in China.

The Problem

The structure of the DNS query caused our team to conclude that a server in the customer production IP space was looking up a home DSL modem in China. This activity is a classic indicator of compromise.

Threat actors are quite fond of compromised home internet connections because the connections are fast, are an awesome place to park stolen goods in transport, and they disguise the attacker.

The ExtraHop analyst asked the customer what the machine was supposed to be doing: The machine's purpose was to prepare content from major entertainment companies for publication.

We could see that Reveal(x) had identified that this machine was also acting as an SSH server, meaning it was receiving inbound SSH traffic.

The analyst asked the customer if the machine was supposed to be an SSH server. The customer said they did have customers who sent files via SFTP. We then used Reveal(x) to look at SSH activity and the clients connecting to this SSH server.

Breach Identified

It looked like someone had opened up a hole in their firewall for a small batch of IP addresses to be able to send in data. We then clicked on the Geomap in Reveal(x) and saw numerous connections from China, some from Taiwan, and just a few connections from the U.S.

This was not an airtight confirmation that something was wrong… but it didn't look good. Another indicator of compromise.

The Investigation

We then looked at an overview of traffic in and out of their system by L7 protocol over the past three days. We saw a steady stream of inbound SSH traffic from IP addresses in China. Not bursty or sporadic, but a constant stream of traffic. Traffic is around 5-15 Kb/s. You'd expect an upload to happen much faster if this was a customer sending files via SFTP. This looked more like human activity.

The customer looked at the logs for the machine in question and saw many instances of "login failed for root." Those errors however, had come to a total stop recently, and the customer's worst fears were confirmed—the bad guys had breached the network.

The Response

The customer completely decommissioned the machine in question and initiated an incident response investigation. A copy of the VM (virtual machine) was then stood up.

The Remediation

It turned out that, rather than allowing inbound connections from a narrow band of IP addresses, somebody had configured the firewall to allow inbound traffic from anywhere on the internet. Misconfigurations are a leading cause of vulnerabilities and, in this case, the results could have been catastrophic. The firewall was reconfigured.

Time Frame

We were able to scope and triage this event in less than 30 minutes.