Walk through the full GSM authentication and encryption flow after the PIN is entered.

The phone registers; the network sends a 128-bit random challenge RAND; the SIM computes SRES = A3(RAND, Ki) to authenticate; both sides derive the session key Kc = A8(RAND, Ki); then A5 encrypts the payload using Kc — and crucially, encryption exists only between handset and base station.

* GSM authentication and Kc generation (A3/A8). — Лихачев Виталий, CC BY-SA 3.0, via Wikimedia Commons. *

Step by step:

0. The PIN is entered by the user (unlocks the SIM)
1. The phone registers (Einbuchen) with the network
2. The network creates a 128-bit RAND (the challenge) and sends RAND to the phone
3. The SIM computes the Signed Response: SRES = A3(RAND, Ki) and sends it back; the network checks A3(RAND, Ki) == SRES
   • → at this point the mobile has authenticated itself to the network
4. Both sides derive the 64-bit connection key: Kc = A8(RAND, Ki)
5. The payload m is encrypted with the A5 stream cipher: A5(m, Kc)

The most security-relevant fact: encryption (and decryption) happens in the BTS — i.e., only on the radio link between the mobile station and the base station. Beyond the base station, into the core and fixed network, the call is not encrypted by this mechanism. So GSM's "encryption" protects the air interface only.

Tip: Note what GSM does NOT do here — the network never proves itself to the phone. Only the phone authenticates. That one-sidedness is the root of the IMSI-catcher problem.

Go deeper:

• Running your own GSM network — Harald Welte & Dieter Spaar (25C3, 2008) — building the network side (BSC/MSC/HLR) from scratch, showing the registration + authentication exchange from the operator's perspective.
• Authentication and Key Agreement (Wikipedia) — the challenge-response pattern GSM pioneered here and that UMTS/LTE/5G later made mutual.