The TCP three-way handshake is the SYN, SYN-ACK, ACK exchange that establishes a reliable connection between two hosts before any data is sent. For security professionals, this handshake is more than networking trivia, it’s the foundation for detecting SYN floods, spotting session hijacking attempts, tuning firewalls, and reading packet captures with confidence. Understanding TCP/IP handshakes for security professionals means knowing not just how the exchange works, but how attackers abuse it and how to detect and defend against those abuses.
Free TCP Handshake Security Cheat Sheet (2026): Want a quick reference while reading? Download the TCP Handshake Security Cheat Sheet for a visual breakdown of the three-way handshake, TCP flags, common attacks, detection tips, and defenses.
TL;DR
- The TCP three-way handshake (SYN → SYN-ACK → ACK) sets up a reliable, ordered connection between a client and a server.
- Each side picks an Initial Sequence Number (ISN) during the handshake. Weak or predictable ISNs enable session hijacking.
- Attackers abuse the handshake through SYN floods, TCP split handshakes, RST injection, and sequence number prediction.
- SYN cookies, rate limiting, stateful firewalls, and IDS/IPS signatures are the core defenses.
- Connection teardown uses a four-way handshake (FIN, ACK, FIN, ACK), which has its own security implications for connection state tracking.
- Wireshark filters like tcp.flags.syn==1 and tcp.flags.ack==0 are essential for spotting handshake abuse in packet captures.
What Is the TCP Three-Way Handshake?
The TCP three-way handshake is the process TCP uses to establish a connection before any application data flows. It exists because TCP is a connection-oriented, reliable protocol, unlike UDP, it needs both sides to agree on starting sequence numbers and confirm the other side is ready to communicate.
What it does:
Synchronizes sequence numbers and confirms bidirectional connectivity.
Why it matters for security:
Every TCP-based attack surface, from port scanning to SYN floods to session hijacking, interacts with this handshake in some way. If you don’t understand the handshake, you can’t fully understand the attack or the defense.
How it works, in one sentence:
The client sends a SYN packet, the server replies with a SYN-ACK, and the client confirms with an ACK, after which the connection is considered “established.”
Why Security Professionals Must Master the TCP Handshake
Understanding TCP/IP handshakes for security professionals isn’t academic, it directly supports core security job functions:
- SOC analysts need to recognize handshake anomalies (half-open connections, malformed flag combinations) in SIEM alerts and packet captures.
- Penetration testers rely on handshake behavior for stealth scanning techniques (SYN scans, idle scans) and for identifying firewall/IDS evasion opportunities.
- Incident responders must distinguish a SYN flood from legitimate traffic spikes during active incidents.
- Network security engineers configure stateful firewalls and load balancers that track handshake state to enforce policy.
- Certification candidates (Security+, CISSP, CEH, CCNA Security) are tested directly on handshake mechanics, TCP flags, and related attacks.
In short: the handshake is a recurring theme across offensive and defensive security work, not a one-time networking lesson.
Penetration testers rely heavily on handshake behavior for stealth scanning and for understanding how exploitation frameworks establish sessions. Once a connection is established, tools like Metasploit take over. Keep this quick-reference guide handy: Metasploit cheat sheet.
Metasploit is the most widely used option, but it is not the only one. A comparison of the leading tools is available here: exploitation frameworks.
Step-by-Step: How the TCP Three-Way Handshake Works
Here’s the exact sequence, including sequence numbers and flags, using a simplified example.
- Step 1: SYN (Client → Server) The client sends a packet with the SYN flag set and a randomly chosen Initial Sequence Number (ISN), for example SEQ=1000.
Client → Server: SYN, SEQ=1000- Step 2: SYN-ACK (Server → Client) The server responds with its own SYN flag, its own ISN (e.g., SEQ=5000), and an ACK acknowledging the client’s sequence number plus one (ACK=1001).
Server → Client: SYN, ACK, SEQ=5000, ACK=1001- Step 3: ACK (Client → Server) The client acknowledges the server’s ISN plus one and the connection moves to the ESTABLISHED state.
Client → Server: ACK, SEQ=1001, ACK=5001Text diagram for packet flow (convert to image later):
CLIENT SERVER
|------ SYN (SEQ=1000) ------->|
|<--- SYN-ACK (SEQ=5000, |
| ACK=1001) --------------|
|------ ACK (ACK=5001) ------->|
| CONNECTION ESTABLISHED |Why the ISN matters
The Initial Sequence Number isn’t arbitrary, it’s a security control. Modern operating systems generate ISNs using cryptographically strong randomization (per RFC 6528) specifically to prevent attackers from predicting sequence numbers and injecting or hijacking packets. Older or misconfigured systems that use predictable ISN generation are vulnerable to session hijacking and blind spoofing attacks.
TCP Flags and Their Security Implications
TCP flags (control bits) determine how a packet behaves and are heavily used in both attacks and defensive filtering.
| Flag | Purpose | Security Relevance |
|---|---|---|
| SYN | Initiates a connection | Abused in SYN floods; used in stealth scans |
| ACK | Acknowledges received data | Missing ACK on data packets can indicate scanning or spoofing |
| RST | Resets/terminates a connection abruptly | Used in RST injection attacks to hijack or kill sessions |
| FIN | Gracefully closes a connection | Used in FIN scans to evade some firewalls/IDS |
| PSH | Pushes buffered data to the application | Rarely security-relevant alone, but useful in fingerprinting |
| URG | Marks urgent data | Historically abused in certain evasion and crafted-packet attacks |
Unusual flag combinations: such as SYN+FIN together, or a full “Christmas tree” packet (SYN, FIN, URG, PSH all set), rarely occur in normal traffic and are strong indicators of reconnaissance or crafted malicious packets.
Common Attacks That Target the TCP Handshake
1. SYN Flood
What it is:
An attacker sends a high volume of SYN packets, often with spoofed source IPs, without ever completing the handshake. The server allocates resources for each half-open connection, exhausting its connection table.
Impact:
Denial of service: legitimate clients can’t establish new connections.
Key indicator:
A large number of connections stuck in the SYN_RECEIVED (half-open) state.
2. ISN Prediction / Session Hijacking
What it is:
If a target’s ISN generation is predictable, an attacker can guess the sequence number sequence and inject packets into an existing TCP session, potentially hijacking an authenticated connection without needing credentials.
Impact:
Unauthorized session takeover, data injection, or man-in-the-middle-style manipulation.
Key indicator:
Duplicate or out-of-window sequence numbers, unexpected RSTs, or session anomalies in logs.
3. TCP Split Handshake
What it is:
A non-standard handshake variation where both hosts send SYN packets simultaneously (a “simultaneous open”), which some firewalls and stateful devices mishandle. Stateful devices are built around an assumption that one side initiates with a lone SYN and the other replies with SYN-ACK; when both endpoints send SYN packets at nearly the same time, the device’s state machine can lose track of which packet started the connection and end up recording the session in an inconsistent or incomplete state. Attackers can exploit that inconsistency; especially across asymmetric network paths, where the firewall only sees part of the traffic to smuggle a connection past inspection without it ever being properly validated.
Impact:
Firewall/IDS evasion, unauthorized connection establishment.
Key indicator:
Simultaneous SYN packets from both endpoints without the expected sequential SYN → SYN-ACK → ACK pattern.
4. RST Injection
What it is:
An attacker spoofs a TCP RST packet with a valid-looking sequence number to forcibly and prematurely terminate a legitimate connection between two parties.
Impact:
Denial of service against specific sessions; historically used for large-scale connection disruption and censorship.
Key indicator:
Unexpected RST packets not preceded by normal application-layer closure behavior, especially from unexpected source addresses.
Beyond sequence validation:
Sequence number checks alone don’t fully solve RST injection, since a sufficiently informed attacker can still land a packet inside the receive window. Two additional mitigations are common in 2026 environments: TCP-AO (Authentication Option, RFC 5925) or its predecessor TCP MD5, which cryptographically sign TCP segments so forged RST packets are rejected outright; and simply moving sensitive sessions to an encrypted transport (TLS/IPsec), where an injected RST can still tear down the TCP connection but can’t manipulate or read the data itself.
Many of the reconnaissance techniques that interact with the TCP handshake (especially SYN scans) are performed with Nmap. For a complete command reference, see this Nmap cheat sheet: Nmap cheat sheet.
When you run Nmap with default or custom scripts, the most useful ones are covered here: Top 10 NSE scripts.
Detecting Handshake Abuse with Wireshark and Logs
Wireshark Analysis
These filters are staples for any security professional analyzing packet captures:
- Detect SYN floods (SYNs without matching ACKs): tcp.flags.syn==1 and tcp.flags.ack==0
- Find half-open connections: tcp.analysis.retransmission and tcp.flags.syn==1
- Spot RST packets: tcp.flags.reset==1
- Identify suspicious flag combinations (e.g., SYN+FIN): tcp.flags.syn==1 and tcp.flags.fin==1
- Filter for a specific conversation to trace full handshake behavior: tcp.stream eq <stream number>
Logs and Stateful Inspection
- Firewall/connection logs: Look for spikes in half-open connections or a high SYN-to-ACK ratio.
- Stateful inspection devices track connection state (NEW, ESTABLISHED, RELATED), a flood of connections stuck in NEW state is a red flag.
- NetFlow/IPFIX data can reveal SYN flood patterns at scale without needing full packet captures.
IDS/IPS
Modern IDS/IPS platforms (Suricata, Snort) include signatures for SYN flood thresholds, malformed flag combinations, and known RST-injection patterns. Tuning these thresholds to your network’s baseline traffic is essential to avoid false positives during legitimate traffic surges.
These Wireshark filters become even more powerful when you have a complete reference of display filters, capture filters, and keyboard shortcuts at hand. Keep this updated Wireshark cheat sheet open while analyzing handshake traffic: Wireshark cheat sheet.
Defenses Against Handshake-Based Attacks
| Attack | Primary Defense |
|---|---|
| SYN flood | SYN cookies, connection rate limiting, SYN queue tuning |
| ISN prediction / hijacking | Cryptographically random ISN generation (RFC 6528), TLS/IPsec for session integrity |
| TCP split handshake | Strict firewall state validation, disabling simultaneous-open acceptance |
| RST injection | TCP sequence number validation, IPsec, encrypted transport (TLS) |
SYN cookies: Instead of storing state for every half-open connection, the server encodes connection information into the SYN-ACK’s sequence number itself. No resources are reserved until the final ACK arrives with a valid cookie, effectively neutralizing SYN flood resource exhaustion.
Rate limiting: Restricting the number of new SYN packets accepted per source IP per second reduces flood impact without disabling legitimate traffic entirely.
Firewalls and load balancers: Modern stateful firewalls and load balancers (e.g., those performing SYN proxying) can absorb and validate the handshake on behalf of backend servers, shielding them from direct exposure.
Sequence number randomization: Ensuring operating systems and network stacks use strong, unpredictable ISN generation closes the door on classic session hijacking techniques.
Modern techniques (2026): Cloud-based DDoS scrubbing services, anycast-based load distribution, and hardware-accelerated SYN cookie generation at the network edge are now standard for large-scale SYN flood mitigation, especially for internet-facing infrastructure.
TCP Handshake vs TLS Handshake
These two handshakes are often confused but operate at different layers and solve different problems.
| Aspect | TCP Handshake | TLS Handshake |
|---|---|---|
| Layer | Transport (Layer 4) | Session/Presentation (Layer 5-6, above TCP) |
| Purpose | Establish reliable connection | Establish encrypted, authenticated session |
| Steps | 3 (SYN, SYN-ACK, ACK) | Varies (1-RTT or 0-RTT in TLS 1.3) |
| Security role | Connection setup only, no encryption | Provides confidentiality, integrity, authentication |
| Runs when | Before any data transfer | After TCP handshake completes (for TLS over TCP) |
Key takeaway:
The TCP handshake gets you connected. The TLS handshake, which happens afterward over that TCP connection, is what actually secures the data. A completed TCP handshake tells you nothing about whether the traffic is encrypted or trustworthy.
Connection Termination: The Four-Way Handshake
TCP connections don’t just stop, they close through a four-way handshake, sometimes called the graceful termination sequence:
- Host A sends FIN (I’m done sending).
- Host B sends ACK (acknowledging A’s FIN).
- Host B sends its own FIN (I’m done sending too).
- Host A sends ACK (acknowledging B’s FIN).
HOST A HOST B
|-------- FIN ---------------->|
|<------- ACK -----------------|
|<------- FIN -----------------|
|-------- ACK ----------------->|
| CONNECTION CLOSED |Security relevance:
Connections can also terminate abruptly via a single RST packet, bypassing the graceful four-way process entirely. Security tools should distinguish normal FIN-based closures from abrupt RST-based terminations, since the latter is far more common in attacks, crashed processes, or scanning activity (e.g., a completed SYN scan that immediately sends RST after receiving SYN-ACK).
Practical Tips for Security Professionals
- Build a home lab: Use two VMs and Wireshark to capture and study a normal handshake before analyzing malicious traffic samples.
- Practice with public PCAP repositories: Analyze known SYN flood and scanning captures to build pattern recognition.
- Correlate flags with tools: Learn what Nmap scan types (-sS SYN scan, -sF FIN scan, -sX Xmas scan) look like at the packet level, this bridges offensive tool usage with defensive detection.
- Baseline your environment: Know your normal SYN-to-ACK ratio and half-open connection count so anomalies stand out immediately during an incident.
- Practice writing IDS rules: Write a simple Suricata/Snort rule to detect an abnormal SYN flag ratio, then test it against sample traffic.
- Study RFC 793 and RFC 6528: These are the foundational documents for TCP state machine behavior and secure ISN generation, required reading for deep technical mastery.
The best way to internalize TCP handshake behavior is to observe it live in a controlled environment. Building your own isolated lab lets you generate SYN floods, practice SYN scans, and watch the three-way handshake in Wireshark without any risk. Start with this step-by-step guide: How to build a home penetration lab. For a broader overview that also covers progressive lab paths, see: How to set up a penetration testing lab.
A quick lab scenario:
Spin up two VMs, call them victim (running a simple web server) and attacker. On attacker, run hping3 -S –flood -p 80 victim-ip to generate a stream of SYN packets with no intention of completing the handshake. On victim, run netstat -an | grep SYN_RECV and watch the count climb, then open Wireshark on the link between them and apply tcp.flags.syn==1 and tcp.flags.ack==0. You’ll see exactly what a SYN flood looks like at the packet level and how it fills the connection table, then repeat the test with SYN cookies enabled on victim and compare the difference. This kind of before/after comparison is one of the fastest ways to internalize why a specific defense works, not just that it works.
Frequently Asked Questions FAQ’s
What is the TCP three-way handshake in simple terms?
It’s the three-step process (SYN, SYN-ACK, ACK) that two computers use to establish a reliable connection before exchanging data.
Why is the Initial Sequence Number (ISN) important for security?
The ISN must be unpredictable. If an attacker can guess it, they can potentially hijack or inject data into an active TCP session.
What is a SYN flood attack?
A SYN flood is a denial-of-service attack where an attacker sends many SYN packets without completing the handshake, exhausting the server’s connection resources with half-open connections.
How do SYN cookies prevent SYN flood attacks?
SYN cookies let a server avoid storing connection state until the final ACK arrives, encoding the necessary information into the SYN-ACK’s sequence number instead, so flooding with incomplete handshakes no longer exhausts resources.
What’s the difference between a TCP handshake and a TLS handshake?
The TCP handshake establishes a basic, unencrypted connection at the transport layer. The TLS handshake happens afterward and adds encryption, integrity, and authentication on top of that connection.
What is a TCP split handshake attack?
It’s a technique that abuses simultaneous-open behavior (both sides sending SYN at once) to potentially bypass firewalls or security devices that don’t correctly validate handshake state.
How many steps are in TCP connection termination?
Four: FIN, ACK, FIN, ACK; known as the four-way handshake. Connections can also end abruptly with a single RST packet.
What Wireshark filter detects a SYN flood?
tcp.flags.syn==1 and tcp.flags.ack==0 helps surface a high volume of SYN packets without corresponding ACKs, a common flood indicator.
Can TCP sequence numbers be predicted?
On modern operating systems using RFC 6528-compliant randomization, prediction is extremely difficult. Legacy or misconfigured systems with weak ISN generation remain vulnerable.
Download the TCP Handshake Security Cheat Sheet
Keep the key concepts from this guide within reach. The TCP Handshake Security Cheat Sheet 2026 summarizes the SYN → SYN-ACK → ACK process, important TCP flags, handshake-based attacks, Wireshark detection tips, and common defenses in one quick-reference resource.
