How are Linux system calls invoked on x86_64, and how does the syscall register convention differ from the normal function ABI?

Put the syscall number in %rax, the arguments in %rdi, %rsi, %rdx, %r10, %r8, %r9, then execute syscall; the result returns in %rax. Note arg 4 is %r10, not %rcx.

* Syscall number and result in rax; arg 4 uses r10 not rcx because the syscall instruction clobbers rcx. *

A system call is a controlled jump into the kernel. Unlike a normal call, you select the operation with a number in %rax and trigger the syscall instruction:

mov  $1, %rax        # syscall 1 = write
mov  $1, %rdi        # fd = 1 (stdout)
lea  msg(%rip), %rsi # buffer
mov  $13, %rdx       # count
syscall              # trap into the kernel

The %r10 quirk: the user-space function ABI uses %rcx for the 4th argument, but the syscall instruction itself clobbers %rcx (it stashes the return address there). So the kernel ABI substitutes %r10 for argument 4. Everything else matches the function ABI.

Note: Normally you don't write this by hand — libc provides wrapper functions (write(), exit()) that set up the registers and issue syscall for you.

Go deeper:

• System call (Wikipedia) — system calls as the kernel interface.
• syscall(2) — Linux manual page — the per-architecture syscall registers and numbers.
• SYSCALL — x86 instruction reference — the SYSCALL instruction itself.