How do you systematically approach an unknown assembly function?

Work in five steps: identify the inputs (argument registers) and output (%rax at ret), map the control flow, trace each path, annotate with pseudocode, then name what it computes.

* Five steps: inputs/output, control flow, trace, pseudocode, name it. *

Step 1 — Identify inputs and outputs:

• What registers are read before being written? Those are arguments (%rdi, %rsi, %rdx, %rcx, %r8, %r9)
• What's in %eax/%rax at ret? That's the return value

Step 2 — Map the control flow:

• Find all jmp, je, jne, etc. — draw arrows
• Identify loops (backward jumps) and branches (forward jumps)
• Find call instructions — note what functions they call

Step 3 — Trace each path:

• Start at the top, follow the "happy path" first
• Then trace what happens when conditions fail

Step 4 — Annotate with pseudocode:

# eax = rdi + rsi*4 → "eax = a + b*4"
# cmp + jl → "if (a < b)"

Step 5 — Name it:

• What does it compute? String compare? Sum? Search?

GDB shortcuts:

(gdb) disas func_name
(gdb) break *func_address
(gdb) info registers
(gdb) x/... $rsp

Go deeper:

• Disassembler - how tools recover mnemonics and control flow from machine code