The Trust Problem: Networks Without Authentication

The internet and local networks are built on protocols designed for communication functionality, not security. A core issue is the lack of built-in authentication in the protocols that enable networking. When your computer sends a packet onto the network, there’s no cryptographic proof of who sent it. When a router announces it can reach certain networks, there’s no verification that the announcement is legitimate. When a device on your local network claims to have a particular IP address, everyone believes it.

This isn’t a bug; it’s a feature, or at least a deliberate design choice. The internet’s architecture follows the end-to-end principle: keep the network core simple and put intelligence at the edges. The network’s job is to move packets from source to destination as efficiently as possible. Everything else (reliability, ordering, security) should be implemented at the endpoints when needed, not built into the network infrastructure itself.

This principle has proven remarkably successful. The Internet Protocol itself doesn’t guarantee reliable delivery, but TCP implements reliability in software at the endpoints. The network doesn’t ensure packets arrive in order, but TCP handles reordering. The original end-to-end paper even discussed security and encryption as functions that should be implemented at the endpoints rather than in the network.

The problem is that many of the application-level protocols we still rely on weren’t designed with security in mind. Protocols like ARP (for mapping IP addresses to hardware addresses), BGP (for internet routing), and DNS (for translating domain names to IP addresses) were created when the internet was a small, trusted community of academic and research institutions. These protocols assumed honest participants. Adding authentication seemed unnecessary and would have introduced complexity.

As networks grew to connect billions of devices and users with varying motivations, these trust assumptions became vulnerabilities. But replacing core protocols is extraordinarily difficult when the internet runs on installed hardware and software across millions of networks. The result is that we have a global network built on protocols that fundamentally trust participants to be honest, being used by untrustworthy actors.

Security has been added as extensions and overlays: TLS for encrypted web traffic, IPsec (along with OpenVPN, IKEv2, and others) for encrypted network traffic, DNSSEC for authenticated DNS. But the core protocols remain largely unauthenticated. Attacks exploit the trust assumptions built into these protocols. Defenses can work around these limitations through configuration, monitoring, and layered protections.

Introduction: Layers of Vulnerability

Networks connect everything, from laptops to industrial systems, locally and around the globe.

To support a variety of underlying hardware and address specific communication challenges, the networking stack is organized in layers, each providing different services and each with its own security vulnerabilities. We’ll examine three critical layers:

The Data Link layer handles communication between devices on the same local network: your computer talking to your router, computers in an office talking to each other through switches. Attacks at this layer exploit the trust assumptions built into local network protocols.

The Network layer handles routing traffic between different networks: how data gets from your local network through multiple routers across the internet to reach its destination. Attacks at this layer can misdirect traffic, overwhelm routers, or intercept communications.

The Transport layer manages end-to-end communication between applications, ensuring data arrives reliably and in order. Attacks at this layer exploit the connection establishment process or the stateless nature of certain protocols.

Data Link Layer: Exploiting Local Network Trust

The Data Link layer operates within your local network: the subnet where your computer, other nearby devices, and your router all communicate directly via an ethernet switch or Wi-Fi access point. At this layer, devices trust that other devices on the same network are legitimate. This trust creates attack opportunities.

CAM Table Overflow: Turning Switches into Hubs

Modern networks use switches to connect devices. When a switch receives network traffic, it needs to know which physical port to send it out. Switches maintain a Content Addressable Memory (CAM) table that maps each device’s MAC address (a unique hardware identifier) to the physical port to which it is connected.

When a switch sees traffic from a new MAC address, it adds that address to its CAM table along with the port where the traffic originated. This allows the switch to intelligently forward traffic only to the relevant port rather than broadcasting it everywhere. This is more efficient and more private than the older network-hub technology, which simply broadcasts all traffic to all ports.

However, CAM tables have a limited size, typically storing a few thousand to tens of thousands of entries. An attacker can exploit this limitation through CAM table overflow. The attacker sends traffic with thousands of fake MAC addresses, flooding the switch’s CAM table. Once the table fills up, the switch can no longer learn where devices are located.

When a switch’s CAM table is full and it receives traffic for an unknown MAC address, it has no choice but to fall back to hub-like behavior: broadcasting the traffic to all ports. Now the attacker, connected to one port, can see all network traffic, even traffic intended for other devices. This defeats the privacy protection that switches normally provide.

The attack is simple to execute. Tools (like macof, part of the dsniff suite) can generate traffic with random MAC addresses at high rates.

Defending Against CAM Overflow

Defense requires managed switches (switches with security features that can be configured, unlike simple unmanaged switches):

• Port security: Limit the number of MAC addresses that can be learned on each port

• MAC address limiting: Set maximum MAC addresses per port (e.g., 5 addresses maximum)

• Sticky MAC addresses: Lock each port to specific known MAC addresses

• Violation actions: Configure what happens when limits are exceeded (shut down port, send alerts, drop packets)

VLAN Hopping: Breaking Network Segmentation

Organizations often divide their networks into Virtual LANs (VLANs) for security and organization. VLANs create logical network segments on the same physical infrastructure. For instance, the accounting department might be on VLAN 10, engineering on VLAN 20, and guest WiFi on VLAN 30. Devices on different VLANs normally cannot communicate directly; traffic must go through a router, where access controls can be enforced.

VLANs work by using tags attached to network frames. When a switch receives tagged traffic, it knows which VLAN the traffic belongs to and forwards it only to ports in that VLAN. This segmentation is meant to provide security isolation.

VLAN hopping attacks bypass this isolation. In a switch spoofing attack, the attacker’s device impersonates a switch by sending special negotiation messages. If the real switch accepts the negotiation, it creates a trunk connection (a special link that carries traffic for all VLANs rather than just one). Now the attacker can send and receive traffic on any VLAN by adding the appropriate VLAN tags.

In a double tagging attack, the attacker sends frames with two VLAN tags. The first switch removes the outer tag and forwards the frame based on it. The next switch sees only the inner tag and forwards based on that. This allows the attacker to inject traffic into a different VLAN than their own.

Defending Against VLAN Hopping

Defense requires proper configuration of managed switches:

• Disable automatic trunk negotiation on user-facing ports (only administrator-configured ports should become trunks)

• Explicitly configure trunk ports: Specify which ports are trunks and which VLANs they carry

• Use native VLAN tagging: Tag all VLAN traffic on trunk ports, including the native VLAN

• Place unused ports in an isolated VLAN: Prevent attackers from connecting rogue devices to unused switch ports

Many VLAN hopping attacks exploit default switch configurations that prioritize ease of setup over security.

ARP Cache Poisoning: Redirecting Local Traffic

When devices on a local network want to communicate, they need to know each other’s hardware addresses. The Address Resolution Protocol (ARP) provides this mapping. When your computer wants to send data to another device on the local network, it broadcasts an ARP request: “Who has IP address 128.6.13.4?” The device with that IP address responds: “I do, and my MAC address is b4:96:91:cf:f4:a4.”

Devices cache these ARP responses to avoid the overhead of repeatedly broadcasting requests and waiting for responses. The problem is that ARP has no authentication. Devices trust any ARP response they receive, even unsolicited ones.

ARP cache poisoning (also called ARP spoofing) exploits this trust. An attacker sends fake ARP responses claiming to have IP addresses of other devices on the network. These can be gratuitous ARPs: unsolicited ARP announcements (i.e., responses that are not associated with a request) that devices commonly send when they boot up to inform the network of their IP-to-MAC mapping. Since gratuitous ARPs are normal and expected, devices accept them without question. An attacker can simply broadcast “I have IP address 192.168.1.1 (the router’s address), and my MAC address is b4:96:91:dc:f4:ad (the attacker’s address).”

Devices that receive this fake response update their ARP caches with the false information. Now, when they try to send traffic to the router, they’re actually sending it to the attacker’s machine. The attacker can then inspect the traffic, modify it, and forward it to the real router. The victim’s computer thinks it’s communicating directly with the router, but everything is flowing through the attacker’s system first.

To avoid outages and stay stealthy, attackers usually enable IP forwarding or run a transparent bridge so traffic continues after redirection.

This is a classic man-in-the-middle attack. The attacker positions themselves between two communicating parties and can intercept, read, or modify all traffic passing between them. Even if the upper-layer protocols use encryption (like HTTPS), the attacker can see which websites are being accessed, the amount of data being transferred, and can potentially launch further attacks.

The attack is particularly effective because most devices have no way of verifying ARP information. If you receive an ARP response, you believe it, even if you didn’t ask for it and even if you already had correct information in your cache.

Defending Against ARP Poisoning

Defense mechanisms include:

• Static ARP entries: Manually configure IP-to-MAC mappings that cannot be overwritten by ARP responses (practical only for critical systems like routers and servers – it’s a real hassle to manage in most environments)

• Dynamic ARP Inspection (DAI): A managed switch feature that validates ARP packets against a trusted database of IP-to-MAC bindings; untrusted ARP responses are dropped

• ARP monitoring tools: Software that detects suspicious ARP activity, such as multiple MAC addresses claiming the same IP address

• Port security: Limits which MAC addresses can appear on each switch port

IPv6 Neighbor Discovery: The Same Attack, Different Protocol

IPv6, the newer version of the Internet Protocol designed to solve IPv4’s address exhaustion problem, replaced ARP with the Neighbor Discovery Protocol (NDP). Unfortunately, NDP has similar security weaknesses.

In IPv6, devices use NDP to discover other devices on the local network. Instead of ARP requests and responses, IPv6 uses Neighbor Solicitation and Neighbor Advertisement messages. A device broadcasts “Who has IPv6 address 2001:db8::5?” and the device with that address responds with its MAC address.

Just like ARP, NDP lacks authentication. An attacker can send fake Neighbor Advertisement messages, poisoning IPv6 neighbor caches the same way ARP caches are poisoned. The attacker claims to own IPv6 addresses of other devices, redirecting traffic through their system for inspection or modification.

The attack techniques and impacts are essentially the same as ARP poisoning, just using IPv6 messages instead of IPv4 ARP.

Defending Against NDP Poisoning

Defense mechanisms parallel those for ARP:

• Static neighbor entries: Manually configure IPv6-to-MAC mappings

• ND Inspection (or ND Snooping): A managed switch feature that validates Neighbor Discovery messages (different vendors use different terms for this feature: it’s also referred to as RA Guard, DHCPv6 Guard, and Source Guard)

• SEcure Neighbor Discovery (SEND): Uses cryptographic signatures to authenticate NDP messages (rarely deployed due to complexity)

The important lesson is that upgrading to IPv6 doesn’t solve the fundamental trust problem in local network protocols. The same architectural vulnerabilities persist in the new protocol.

DHCP Spoofing: Controlling Network Configuration

When devices connect to a network, they typically don’t have their IP addresses pre-configured or other important information for that’s needed to function on the network. Instead, they use the Dynamic Host Configuration Protocol (DHCP) to automatically obtain network settings. A device broadcasts “I need an IP address,” and a DHCP server responds with an IP address to use, the subnet mask, the default gateway (router), and addresses of DNS servers.

DHCP spoofing involves an attacker running a rogue DHCP server on the network. When legitimate devices request network configuration, both the legitimate DHCP server and the attacker’s rogue server respond. If the victim device accepts the attacker’s response first, it configures its network settings according to the attacker’s specifications. The attacker can gain an advantage by deploying a denial-of-service attack on the real DHCP server to slow it down or completely stop it from responding.

The attacker can provide a legitimate-looking IP address and subnet mask but specify their own machine as the default gateway. Now all traffic from the victim device attempting to reach outside the local network goes through the attacker’s system first. The attacker can also specify malicious DNS servers, allowing them to redirect web traffic by providing false IP addresses for websites.

This attack enables comprehensive man-in-the-middle capabilities. Every connection the victim makes to the internet flows through the attacker’s machine. The attacker can intercept credentials, inject malicious content, or simply monitor all activity.

Besides spoofing, an attacker can perform DHCP starvation by requesting many leases to exhaust the pool.

Defending Against DHCP Spoofing

Defense requires managed switches with DHCP security features:

• DHCP snooping: Designate specific ports on the switch as trusted for DHCP server responses; responses from untrusted ports are blocked

• DHCP binding table: Switch maintains a table mapping IP addresses to MAC addresses to ports, validating that subsequent traffic matches these bindings

• Port security: Restrict which MAC addresses are allowed on each port, preventing unauthorized devices from connecting

• Rate limiting: Limit the rate of DHCP requests from each port to prevent DHCP exhaustion attacks

Common Threads in Data Link Attacks

Several patterns emerge across these Data Link layer attacks:

Trust without verification: Protocols assume devices on the local network are legitimate and tell the truth. ARP, NDP, and DHCP all lack authentication mechanisms.

Broadcast-based discovery: Many local network protocols use broadcasts to find services and devices. Attackers can listen to these broadcasts and respond maliciously.

Limited scope: These attacks require the attacker to be on the local network. Remote attackers on the internet cannot execute ARP poisoning or DHCP spoofing against your local network. However, an attacker who gains access to the local network (through WiFi, physical access, or compromising any device on the network) can then launch these attacks against other local devices.

Man-in-the-middle positioning: Most of these attacks enable the attacker to position themselves between the victim and legitimate network resources, allowing interception and modification of traffic.

Defense generally requires moving from implicit trust to verification: using switch features that validate protocol messages, restrict which devices can provide network services, and monitor for suspicious activity.

Network Layer: Routing and Misdirection

The Network layer handles routing: determining the path traffic should take from source to destination across multiple networks. This layer is where the Internet Protocol (IP) operates, along with routing protocols that allow routers to share information about network topology.

Router Vulnerabilities

Routers are critical network infrastructure. They receive traffic and forward it toward its destination based on routing tables that map IP address ranges to next-hop destinations. Compromised routers can disrupt entire networks or enable large-scale traffic interception.

A router is a special-purpose computer. Like a computer, it runs an operating system, has an administrative interface that’s protected with a login and password, and often provides additional services beyond routing, such as network address translation (NAT), DNS, quality of service prioritization, network management, and time synchronization.

Denial of Service (DoS) attacks against routers aim to overwhelm them with traffic or requests, preventing them from handling legitimate traffic. Routers have limited CPU and memory resources. An attacker might send floods of traffic that the router must process, consuming its resources. Or the attacker might exploit protocol weaknesses that cause the router to consume excessive resources when handling certain types of traffic.

When routers are overwhelmed, they drop packets. Legitimate traffic cannot get through. If the router is a bottleneck connecting networks, the attack effectively disconnects those networks from each other or from the internet.

Route table poisoning involves injecting false information into a router’s routing table. If an attacker can access the router’s management interface (either through default credentials, unpatched vulnerabilities, or network access to management ports), they can modify routing tables directly. Alternatively, if the attacker can inject malicious routing protocol messages, they might trick the router into learning incorrect routes.

By poisoning routing tables, the attacker can redirect traffic. Setting routes for certain destinations to point to the attacker’s network allows them to intercept traffic intended for those destinations. Setting routes to point to non-existent destinations creates black holes where traffic disappears.

Insecure router configurations create many vulnerabilities. Many routers ship with default administrative credentials (username: admin, password: admin). Various websites keep track of these defaults across different vendors and models. For example, portforward.com, router-network.com, and routerpasswords.com.

Administrators who don’t change these defaults leave routers vulnerable to anyone who knows the common defaults. Routers with management interfaces exposed to the internet (rather than restricted to internal management networks) are exposed to attack from anywhere. Outdated firmware leaves routers vulnerable to known exploits.

The Mirai botnet (created by a Rutgers student, which we discussed earlier in the malware lecture) demonstrated the scale of insecure router (and other IoT device) compromise. By scanning the internet for devices with default credentials and exploiting common vulnerabilities, Mirai compromised hundreds of thousands of routers and used them to launch devastating DDoS attacks.

Defense against router attacks

Defense requires:

• changing default credentials

• restricting management interface access to trusted networks only,

• keeping firmware updated,

• disabling unnecessary services

• monitoring for suspicious routing changes.

Many home and small business routers have weak security practices, putting the organization’s infrastructure at risk and creating internet-wide vulnerabilities.

BGP Hijacking: Misdirecting Internet Traffic

The internet is a network of networks. Autonomous Systems (typically Internet Service Providers or large organizations) connect to each other and exchange routing information using the Border Gateway Protocol (BGP). BGP allows these systems to announce which IP address ranges they can reach and how to get to them.

Here’s the problem: BGP has no authentication. When one AS announces via BGP that it can reach certain IP addresses, other systems generally believe the announcement and update their routing tables accordingly. This trust enables the internet to work at scale, but it also creates a massive vulnerability.

BGP hijacking occurs when a malicious or misconfigured AS announces IP address ranges it doesn’t actually own. Suppose an attacker controls an AS and announces that they have the best route to reach 1.2.3.0/24 (256 IP addresses from 1.2.3.0 to 1.2.3.255). Other ASes that receive this announcement will start sending traffic for those addresses to the attacker’s AS.

Now, all traffic intended for those IP addresses flows to the attacker. The attacker can:

• Black hole the traffic: Simply drop it, causing a denial of service for the victims

• Intercept the traffic: Inspect it for valuable information, then forward it to the legitimate destination (man-in-the-middle)

• Impersonate services: Respond to connection attempts as if they were the legitimate services at those IP addresses

BGP hijacking has been used for various attacks. For example:

• In 2008, Pakistan Telecom accidentally hijacked YouTube’s IP addresses while attempting to block YouTube domestically, causing YouTube to become unreachable worldwide for several hours (we’ll discuss this example in more detail).

• In 2018, attackers hijacked routes to Amazon Route 53 (a DNS service) IP addresses, so victims received malicious DNS answers for myetherwallet.com. Users were presented with a TLS warning, but many clicked through.

• In the 2022 KLAYswap attack, attackers hijacked a prefix to a content delivery domain used by KLAYswap’s web app and served malicious JavaScript to users.

The fundamental issue is that BGP was designed for a small, trusted community of network operators. In the modern internet with thousands of ASes operated by diverse organizations with varying security practices, this trust model is inadequate.

Defending Against BGP Hijacking

Defense mechanisms are being deployed but adoption is slow:

RPKI (Resource Public Key Infrastructure) creates a hierarchical system of cryptographic certificates that prove ownership of IP address ranges:

• Regional Internet Registries (like ARIN, RIPE) issue certificates to organizations that own IP address blocks

• Organizations create Route Origin Authorizations (ROAs) that specify which AS is authorized to announce their IP addresses

• ROAs are cryptographically signed using public key cryptography

• When an AS receives a BGP announcement, it can validate the announcement against the ROA

• If someone tries to announce IP addresses they don’t own, the signature won’t match and the announcement can be rejected

RPKI provides proof of authorization to announce routes but doesn’t protect the BGP messages themselves during transit.

BGPsec adds cryptographic authentication to the BGP path itself:

• Each AS in the path digitally signs its portion of the route announcement

• The signature includes the AS number, the IP prefix, and the next AS in the path

• Signatures form a chain: AS1 signs and passes to AS2, AS2 signs and passes to AS3, etc.

• Receiving ASes can verify the entire chain of signatures

• This proves that each AS in the advertised path actually participated in forwarding the announcement

BGPsec prevents attackers from forging routes or modifying the AS path. However, it requires more computational resources and changes to BGP implementations.

BGP route filtering uses manually configured policies about which announcements to accept:

• Filter based on expected IP ranges from each neighbor

• Reject announcements for your own IP addresses coming from external sources

• Implement prefix length limits (reject /24 announcements if you expect /20)

BGP monitoring services detect suspicious announcements by monitoring global BGP routing tables and alerting operators to unexpected changes, like suddenly seeing announcements for address ranges from unexpected sources.

However, BGP security improvements require cooperation across the internet ecosystem. Individual organizations can follow best practices for their own networks, but protecting against BGP hijacking at internet scale requires widespread adoption of new protocols and validation practices.

Network Layer Themes

Network layer attacks exploit the distributed trust model of routing. Routers must accept routing information from other routers and make forwarding decisions based on it. When routers are compromised or routing information is manipulated, traffic can be misdirected at scale.

The lack of authentication in routing protocols reflects their design origins in a smaller, more trusted environment. Adding authentication to protocols like BGP is technically possible but requires infrastructure, coordination, and performance considerations that have slowed adoption.

Visibility and detection are challenging. BGP hijacking can be difficult to detect, especially for short-duration attacks. Traffic just appears to be taking a different path through the internet, which happens legitimately all the time as network conditions change.

Transport Layer: Managing Connections

The Transport layer manages end-to-end communication between applications. The two primary protocols at this layer are TCP (Transmission Control Protocol), which provides reliable, ordered delivery, and UDP (User Datagram Protocol), which provides unreliable, unordered delivery. Each has distinct security characteristics.

TCP: The Connection-Oriented Protocol

TCP establishes connections between communicating endpoints and ensures data arrives reliably and in order. This reliability comes from TCP’s connection management mechanisms, but these same mechanisms create attack surfaces.

TCP Connection Establishment: The Three-Way Handshake

TCP uses a three-way handshake to establish connections:

1. SYN: The client sends a SYN (synchronize) packet to the server, including an Initial Sequence Number (ISN) that will be used to track data in this connection

2. SYN-ACK: The server responds with a SYN-ACK packet, acknowledging the client’s ISN and providing its own ISN

3. ACK: The client sends an ACK packet, acknowledging the server’s ISN

After this handshake, the connection is established and data transfer can begin. The sequence numbers allow TCP to detect lost, duplicated, or out-of-order packets and to reassemble data correctly.

Predictable Sequence Numbers: Session Hijacking

Early TCP implementations used predictable Initial Sequence Numbers (ISN). The ISN might simply increment by a fixed amount for each new connection, or be based on an easily-guessed timestamp. This predictability enabled TCP session hijacking.

If an attacker can predict the ISN that will be used in a TCP connection, they can inject packets that appear to be part of that connection. The attacker observes a connection being established between two parties, predicts the sequence numbers being used, and sends packets with those sequence numbers. If the attacker guesses correctly, the packets are accepted as legitimate parts of the connection.

This allows the attacker to inject malicious data into the connection or hijack it entirely, causing the server to believe commands are coming from the legitimate client. Kevin Mitnick’s famous 1994 attack against Tsutomu Shimomura exploited predictable TCP sequence numbers to gain unauthorized access.

Modern TCP implementations use random Initial Sequence Numbers (RFC 6528). Instead of predictable patterns, each new connection uses an ISN chosen randomly from a large space of possible values. This makes predicting the ISN essentially impossible, preventing sequence number-based hijacking attacks.

However, attackers on the same network can still observe TCP connections directly and see the actual sequence numbers being used. Defense against local attackers requires encryption (like TLS/SSL) to prevent meaningful injection even if packets can be forged.

SYN Flooding: Overwhelming the Handshake

The TCP handshake creates an imbalance in resource consumption. When a server receives a SYN packet, it allocates kernel resources for the new connection: memory to track connection state, including the client’s ISN and the server’s chosen ISN. The server then sends SYN-ACK and waits for the client’s ACK to complete the handshake.

If the final ACK never arrives, the server keeps these resources allocated for a timeout period (typically 30-60 seconds) before giving up and freeing them. The server must maintain this half-open connection state because, from its perspective, the ACK packet might just be delayed.

SYN flooding exploits this asymmetry. An attacker sends thousands or millions of SYN packets to the server, typically with spoofed source IP addresses. The server receives each SYN, allocates resources for a connection, sends SYN-ACK (to the spoofed address, where it will be ignored), and waits. The server’s memory fills with half-open connections waiting for ACK packets that will never arrive.

Eventually, the server runs out of resources. It cannot accept new legitimate connections because it has no memory available to track them. Legitimate users trying to connect receive no response; the server is overwhelmed by handling fake connection attempts.

SYN flooding is particularly effective because:

• Sending SYN packets requires minimal attacker resources

• Each SYN causes the server to use substantial resources

• The attacker can spoof source addresses, making the attack traffic difficult to filter

• Even after the server times out and cleans up a half-open connection, the attacker can immediately create a new one

SYN cookies provide an elegant defense mechanism. Instead of allocating resources when receiving a SYN, the server encodes all the necessary connection information into the ISN it sends back in the SYN-ACK. The ISN is computed as a short (32-bit) cryptographic hash of the connection parameters (source IP address, source port, destination IP address, destination port, and a timestamp) combined with a secret key known only to the server.

When the client sends the ACK completing the handshake, it includes this ISN (incremented by one, per TCP protocol). The server can then recompute the hash using the connection parameters from the ACK packet and its secret key. If the recomputed value matches the ISN in the ACK, the server knows this is a legitimate completion of a handshake it initiated. Only then does it allocate resources for the connection. If the ACK never arrives to complete the connection setup (as with SYN flood packets), the server has nothing to clean up.

The tradeoff is that SYN cookies disable some TCP options during connection establishment. However, this is a reasonable compromise: it’s better to have functional connections without some optional features than to be completely offline due to SYN flooding.

Most modern operating systems support SYN cookies, typically activating them automatically when the system detects it’s under SYN flood attack. Rate limiting (restricting how many SYN packets are accepted from a given source) and SYN proxies (intermediate systems that complete handshakes on behalf of servers, only forwarding established connections) provide additional defense layers.

TCP RESET Attacks: Terminating Connections

Once a TCP connection is established, either endpoint can terminate it by sending a packet with the RST (reset) flag set. When a system receives a valid RST packet, it immediately closes the connection without going through the normal graceful shutdown process. This mechanism exists to handle error conditions. If a system crashes and reboots, it can send RST to any connection attempts referring to now-invalid previous connections.

TCP reset attacks exploit this mechanism to forcibly terminate legitimate connections. An attacker who can observe a TCP connection (by being on the same network segment or controlling a router in the path) can inject forged RST packets that appear to come from one of the connection endpoints.

For the reset to be accepted, the attacker must forge several fields correctly:

• Source and destination IP addresses and ports: The attacker observes these from the actual connection traffic

• Sequence number: The RST must have a sequence number within the current receive window of the connection

Early TCP implementations had large receive windows, making it relatively easy for attackers to guess valid sequence numbers. The attacker could send multiple RST packets with different sequence numbers, and at least one would likely fall within the window and be accepted.

The impact is that the connection terminates. If the connection was downloading a file, the download stops. If it was an SSH session, the session drops. If it was carrying VoIP audio, the call ends. Users see connection failures and must reconnect, but may not realize their connection was actively attacked.

TCP reset attacks have been used for censorship. By monitoring network traffic for connections to forbidden sites or for forbidden keywords in traffic, censorship systems can inject RST packets to terminate those connections. China’s “Great Firewall” has historically used TCP reset injection to block access to certain websites. When users attempt to access forbidden content, the system injects RST packets to both client and server, terminating the connection.

Defense against TCP reset attacks has improved through protocol enhancements. Smaller TCP windows (accepting a more restricted range of sequence numbers) make it harder to guess valid sequence numbers. Modern TCP implementations (RFC 5961) use tighter validation rules for RSTs and challenge ACKs, reducing the likelihood of RST injection attacks.

Encryption (TLS/SSL) prevents attackers from seeing the content of connections, making it harder to selectively target connections based on content (though they can still target based on destination or timing patterns).

TCP MD5 signatures (RFC 2385) allow TCP connections to cryptographically authenticate all segments, including RST packets. However, this requires pre-shared keys between endpoints and is primarily used for critical infrastructure like BGP sessions between routers rather than general internet traffic.

The fundamental challenge is that TCP was designed under the assumption that endpoints could trust all packets appearing to be part of their connections. An attacker with network visibility can forge convincing packets. Complete protection requires either encryption (preventing attackers from seeing enough information to forge packets) or cryptographic authentication (proving packets come from legitimate endpoints).

UDP: The Stateless Protocol

UDP is fundamentally different from TCP. It provides no connections, no reliability, no ordering: just simple message delivery. You send a UDP datagram, and maybe it arrives at the destination, maybe it doesn’t. UDP is used when low latency is more important than reliability (like video streaming or online gaming) or for simple request-response protocols (like DNS).

The stateless nature of UDP creates different security characteristics. There’s no handshake to flood, no sequence numbers to predict, no connection state to hijack. However, the lack of connection state means there’s also no way to verify that UDP traffic is part of a legitimate ongoing conversation.

UDP Masquerading: Spoofing Made Simple

UDP provides no authentication of source addresses. An attacker can send UDP packets with any source IP address they choose. This IP address spoofing is possible with TCP too, but TCP’s handshake makes spoofing less useful. If you spoof your source address, the SYN-ACK goes to the spoofed address, not to you, so you can’t complete the connection.

With UDP, there’s no handshake. You send a packet and optionally receive a response. If an attacker spoofs the source address, the response goes to the spoofed address, but for many attack scenarios, the attacker doesn’t care about receiving responses.

Spoofed UDP packets enable several types of attacks. An attacker might send malicious commands to UDP-based services while making it appear the commands came from a trusted source. Or an attacker might trigger responses from services to be sent to a victim’s address, overwhelming them with unwanted traffic (amplification attacks, which we’ll explore in detail when discussing DDoS).

Defense requires network-level controls. BCP 38 (Best Current Practice 38) recommends that ISPs implement ingress filtering, blocking outgoing traffic with source addresses that couldn’t legitimately originate from their network. If your ISP implements this, you cannot send spoofed packets; they get filtered at the ISP boundary. However, BCP 38 adoption is incomplete. Many networks allow spoofed traffic, either due to configuration challenges or lack of incentive (the spoofing enables attacks against others, not against the network itself).

Application-layer protocols running over UDP can add their own authentication to verify message sources, but the underlying UDP protocol provides no protection.

Transport Layer Security Principles

Transport layer attacks often exploit the asymmetry between attackers and defenders. SYN flooding works because sending SYN packets is cheap while handling them is expensive. Amplification attacks work because small queries generate large responses.

Protocol design assumptions create vulnerabilities. TCP’s handshake assumes participants are legitimate. UDP’s statelessness assumes the network will filter spoofed traffic. When these assumptions don’t hold, attacks become possible.

Defense requires multiple layers. No single defense solves these problems. SYN cookies plus rate limiting plus connection tracking provides better protection than any single mechanism. Anti-spoofing at network edges plus rate limiting at servers plus filtering at victims creates defense in depth.

The Interconnected Nature of Network Security

Network security challenges span multiple layers, and vulnerabilities at one layer often enable or amplify attacks at other layers. An attacker who achieves man-in-the-middle positioning through ARP poisoning can then launch more sophisticated application-layer attacks. BGP hijacking enables massive-scale man-in-the-middle attacks that would be impossible at the data link layer alone.

Several themes emerge across all network layers:

Trust without verification: Many protocols were designed assuming participants are honest. ARP, NDP, DHCP, and BGP all operate on trust. Retrofitting authentication and verification is technically possible but faces deployment challenges due to the need for coordination across many systems and organizations.

Asymmetric resource consumption: Attackers exploit situations where their resources consumed (sending spoofed packets, generating SYN floods) are far less than the defender’s resources consumed (maintaining state, processing floods, handling amplified traffic). This asymmetry makes attacks economically favorable.

The visibility problem: Network attacks can be difficult to detect. Is this BGP route change legitimate network optimization or malicious hijacking? Is this traffic flood a DDoS attack or legitimate flash crowd? Distinguishing malicious from legitimate activity requires context, monitoring, and often manual analysis.

Defense in depth: No single defense mechanism is sufficient. Network security requires multiple overlapping protections: switch security features, router configurations, edge filtering, protocol improvements, monitoring, and incident response. Each layer catches some attacks; collectively they raise the bar significantly.

The legacy protocol problem: The internet runs on protocols designed decades ago for a different environment. Fixing fundamental security issues often requires deploying new protocols or protocol extensions across millions of systems operated by thousands of independent organizations. This creates enormous inertia. We know how to improve security (RPKI for BGP, authenticated ARP alternatives), but deployment is slow because it requires coordination, resources, and compatible software across the entire ecosystem.

Network security is ultimately about managing risk in an environment built on trust and openness. Perfect security would require authenticating every message, but this would also make networks rigid, complex, and slow. The challenge is finding the right balance: enough security to mitigate serious threats without undermining the functionality and flexibility that make networks useful.

As networks continue to evolve, new vulnerabilities will emerge. IPv6 brought new variants of old attacks. The Internet of Things creates millions of new network-connected devices with weak security. 5G networks introduce new infrastructure with new attack surfaces. Understanding the fundamental patterns of network attacks (trust exploitation, resource asymmetry, protocol weakness) helps reason about both current and future threats.

The ongoing challenge is to secure a global network infrastructure that was built for openness and interoperability while under constant attack from adversaries ranging from mischievous individuals to sophisticated criminal organizations to nation-states. Network security requires technical solutions, operational practices, and cooperation across organizational and national boundaries. It’s a problem that will never be fully solved, only continuously managed and improved.