How to Identify Malicious Network and Port Scanning

What is Port Scanning?

Network scanning and port scanning—processes for learning about a network's structure and behavior—aren't inherently hostile, but bad actors often use them to conduct reconnaissance before trying to breach a network. Attacks like SUNBURST can use network scanning to get the lay of the land early on in the attack.

In many cases, however, port scans are legitimate processes used to determine the services available on a remote machine and to find which ports on a network are open and may be sending or receiving data. Security analysts scan ports by sending packets and analyzing responses to find vulnerabilities.

What is Malicious Port Scanning?

Port scanning has become an especially useful tool for attackers looking to identify and exploit encrypted network protocols such as Kerberos and SMBv3, which allows them to obscure their actions.

Malicious port scanning can determine what firewalls or other security devices a targeted organization uses, and it can provide other details about the targeted systems, such as identifying computers that are online and applications that are running.

How does malicious scanning work, and how can you detect potential reconnaissance of your network? Let's start by defining the terms at their most basic:

• Network scanning involves detecting all active hosts on a network and mapping them to their IP addresses.
• Port scanning refers to the process of sending packets to specific ports on a host and analyzing the responses to learn details about its running services or locate potential vulnerabilities.

Methods of Network Scanning for Host Discovery

Host discovery, the process of determining what systems on a network are up and listening, is often the first step in a hostile network scanning action. Two protocols are most commonly used for host discovery: Address resolution protocol (ARP) scans and several types of internet control message protocol (ICMP) scans.

Since individual ARP requests are used to map IP addresses to MAC addresses on a local subnet, ARP requests can be sent out to many IP addresses on a local area network (LAN) to determine which hosts are up based on the ones that respond with an ARP reply.

For network scanning outside of a local subnet, several types of ICMP packets can be used instead, including echo, timestamp, and address mask requests. Echo (or ping) requests are used to detect if another host can be reached, while timestamp messages determine the latency between two hosts. Address mask requests are intended to discover the subnet mask in use on the network.

Host discovery for each ICMP message type depends on receiving a corresponding reply from available hosts. If no response is received, it means either that there is no host listening at that address, that the request packet was blocked by a firewall or packet filter, or that the message type isn't supported by the destination device. ICMP echo requests that originate outside an internal network are commonly blocked by firewalls, but timestamp and address mask requests are less likely to be blocked.

Methods of Port Scanning

Once available hosts on a network have been found via networking scanning, port scanning can be used to discover the services in use on specific ports. In general, port scanning attempts to classify ports into one of three designations:

• Open: the destination responds with a packet indicating it is listening on that port, which also indicates that whatever service was used for the scan (commonly TCP or UDP) is in use as well
• Closed: the destination received the request packet but responds with a reply indicating that there is no service listening at the port
• Filtered: the port might be open, but the packet has been filtered out by a firewall and dropped, so no reply is received

Types of TCP Scans

As previously mentioned, TCP and UDP are frequently the protocols used in port scanning. There are several methods of performing TCP scans:

SYN scans, the most common form of TCP scanning, involve establishing a half-open connection to the target port by sending a SYN packet and evaluating the response. The host replies by sending a SYN/ACK packet if the port is open or a RST response if the port is closed. It is also possible for a closed port to reply with an ICMP port unreachable message instead of a RST packet, though this is less common. A lack of any reply indicates that the port is filtered.

A higher level method of TCP scanning is the TCP connect scan, in which the scanner tries to connect to a port via TCP using the connect system call and the full TCP handshake process. This method is utilized less often than SYN scanning, since it requires more overhead in terms of packets and time and is more easily detectable.

NULL, FIN, and Xmas scans are three scan types that involve manipulating TCP header flags. Each of them results in a RST (or ICMP port unreachable) packet from a closed port and no response from an open or filtered port, and they require that the SYN, RST, and ACK bits are not set. NULL scans send packets with no flags set in their headers, while FIN scans have only the FIN bit set. Xmas scan packets, so called because their headers are reminiscent of being "lit up like a Christmas tree," have the FIN, PSH, and URG flag bits turned on.

UDP Scanning

Like TCP scans, UDP scans send a UDP packet to various ports on a target system and evaluate the response to determine the availability of the service and the host. Receiving a UDP packet in response indicates that the port is open, while an ICMP port unreachable error response signifies a closed port. If no response is received, the port could either be open or filtered by a firewall or packet filter.

How to Detect Network and Port Scans

Scan detection methods range from monitoring for simple thresholds and patterns, such as the number of ports connected to from a single origin over a period of time, to probabilistic models based on expected network behavior. Network intrusion detection systems and firewalls are usually configured to detect scans, but scanners can attempt to avoid some common detection rules by altering their scanning rate, accessing ports out of order, or spoofing their source address.

If you don't have the ability to easily track and correlate suspicious behaviors like malicious port scanning, allowing a machine learning solution to establish a baseline of normal network activity can help security teams detect unusual port scanning activity for high-fidelity threat detection.