TCP/IP Isn't Magic. It's Just Lying to You.

Here's the thing about TCP. It doesn't actually guarantee your data arrives. It guarantees it will keep trying until it gives up or you close the connection. That distinction has killed more production systems than I can count.

Start at the bottom or get lost later

IP is dumb. Intentionally dumb. Every packet is a stranger. IP doesn't know if you sent 50 packets before it. It doesn't care. It just looks at the destination address and throws the packet into the network like a dart. Some packets take different routes. Some arrive out of order. Some get dropped entirely because a router buffer filled up for 3 milliseconds at 2am.

IP's job is delivery attempts, not delivery guarantees.

TCP sits on top and plays cleanup. It adds sequence numbers to every segment so the receiving side can reorder them. It adds acknowledgements so the sender knows what got through. It adds a retransmission timer so if an ACK doesn't come back, the segment gets resent. This is the handshake everyone's heard of and almost nobody actually understands.

The three-way handshake is a contract with hidden clauses

SYN, SYN-ACK, ACK. You know the words. Here's what actually happens. Your client sends a SYN with an Initial Sequence Number, a random number your OS picks. The server sends back SYN-ACK with its own ISN and an acknowledgement of yours. Your client ACKs the server's ISN. Now both sides have agreed on two independent sequence number streams, one per direction.

Why random ISNs? Because in the early internet, predictable ISNs let attackers inject packets into your TCP stream without being on-path. This was a real attack. RFC 6528 standardized the fix. Your OS has been doing this quietly for decades.

The connection isn't free. Each half-open SYN consumes memory on the server before the handshake completes. SYN flood attacks exploit exactly this. You can burn through a server's connection table with nothing but unanswered SYNs. We had a scraper hit a payment service in 2021, 60 engineers, Series B company, and it took down checkout for 4 hours on a Friday night before anyone realized it wasn't a code deploy. $340k in lost GMV. The fix was SYN cookies, which had existed since 1996.

Flow control vs congestion control and why you're confusing them

Flow control is about the receiver. The TCP header has a window size field. The receiver advertises how much buffer space it has left. If your receive buffer fills up, window size drops to zero, sender stops. This prevents a fast sender from drowning a slow receiver.

Congestion control is about the network. TCP has no idea how much bandwidth is available between two endpoints. So it guesses. Slow start, AIMD, cubic, BBR. These are algorithms for probing available bandwidth without destroying the network. BBR, which Google shipped in Linux 4.9 around 2016, changed how we think about this. It models bottleneck bandwidth and RTT instead of inferring congestion from packet loss. On a GCP-to-GCP connection we saw 40% throughput improvement switching to BBR for bulk transfers. Same hardware. Same code. Just a kernel flag.

What nobody tells you about TIME_WAIT

Don't do this: restart your service and wonder why you can't bind to port 8080. That's TIME_WAIT. After a connection closes, the side that sent FIN first holds the socket in TIME_WAIT for 2x the Maximum Segment Lifetime, usually 60 seconds on Linux. It exists to absorb any late-arriving packets from the dead connection so they don't corrupt a new one on the same port tuple.

I've seen engineers set SO_REUSEADDR and move on without knowing why it worked. It works because it lets a new socket bind while the old one is in TIME_WAIT, accepting the small risk of a ghost packet arriving. On a high-throughput service doing 50k connections per second you'll exhaust the ephemeral port range before TIME_WAIT clears. We hit this at Netflix on an internal RPC mesh. ss -s showed 30k TIME_WAIT sockets. The answer was connection pooling, not SO_REUSEADDR everywhere.

TCP is reliable the same way a retry loop is reliable. Eventually consistent. Finite patience. Build your intuition around that and you'll stop being surprised by what breaks at 3am.