How Buffer Overflows Actually Hijack Program Execution

Why the Stack Matters

When a program runs, its memory splits into regions: the text segment (executable code), the data segment (globals/statics), the heap (grows upward, dynamically allocated), and the stack (grows downward, holds function call frames).

Every function call pushes a new stack frame containing local variables, the saved base pointer (EBP/RBP), and the return address — the address execution jumps back to once the function finishes. That return address is the target of a classic buffer overflow.

The Vulnerable Pattern

void vulnerable(char *input) {
    char buffer[64];
    strcpy(buffer, input);
}

strcpy copies input into buffer with no length check. If input is longer than 64 bytes, the extra bytes spill past the buffer into whatever sits next on the stack — which, sequentially, is the saved base pointer and then the return address. Write enough carefully chosen bytes and you're no longer overflowing a buffer, you're rewriting where the CPU jumps when the function returns.

How the Exploit Actually Works

A stack-based overflow exploit generally follows five steps:

1. Identify the vulnerability — find a function that copies user input into a fixed-size buffer without bounds checking.
2. Determine the offset — figure out exactly how many bytes are needed before you start overwriting the return address. Tools like Metasploit's pattern_create and pattern_offset generate unique byte patterns so you can pinpoint this precisely instead of guessing.
3. Control the return address — overwrite it with a pointer to memory you control.
4. Insert shellcode — place executable machine code in the buffer itself, so there's something worth jumping to.
5. Use a NOP sled — prepend the shellcode with a run of no-operation instructions. This widens the target area so minor address miscalculations still land inside the sled and slide into the shellcode.

The Defenses That Exist Now

Modern systems don't leave this wide open. Several protections stack on top of each other:

Notice the pattern: almost every defense is eventually worked around with either an information leak or ROP — chaining together small existing code fragments already marked executable, instead of injecting new shellcode. Defense in depth here means each protection raises the cost of an exploit, not that any single one is unbeatable alone.

Heap Overflows Are a Different Beast

Stack overflows go after the return address. Heap overflows instead corrupt the metadata that malloc and free use internally to track allocated chunks. Corrupting that bookkeeping data can hand an attacker an arbitrary write primitive — the ability to write attacker-chosen values to attacker-chosen addresses. It's more complex to pull off than a stack overflow, but it can sidestep protections that are specific to the stack.

Fixing It at the Source

All of this traces back to one root cause: functions that copy data without checking length. The fix starts in the code, not in the mitigations:

• Replace strcpy with strncpy or strlcpy
• Replace sprintf with snprintf
• Replace gets with fgets (never use gets, full stop)
• Where you can, use a memory-safe language like Rust, Go, or Java instead of C/C++

And turn on the compiler-level protections wherever the code does stay in C/C++:

-fstack-protector-all
-D_FORTIFY_SOURCE=2
-pie
-z relro -z now

None of these make unsafe code safe on their own — they raise the bar for exploitation while the real fix is bounds-checked input handling.

Go Deeper

This covers the mechanics of memory layout, exploitation steps, and mitigations. The full lesson is free: Buffer Overflows.