Why doesn't training-time privacy (e.g. DP-SGD) make an LLM secure, and what is "inference-time security"?

Training-time privacy stops the model from memorizing data, but a perfectly trained model can still be attacked while it runs — inference-time security defends the live system against malicious prompts and leaky outputs.

LLM risk lives at two separate attack surfaces, and defending one does nothing for the other:

• Training time — the danger is memorization: the model absorbs verbatim records. The defense is techniques like DP-SGD (Differentially Private Stochastic Gradient Descent) — ordinary SGD is the optimizer that nudges the weights on each training batch; the "differentially private" variant clips each example's gradient and adds calibrated noise, so no single training record can move the weights much and secrets don't get baked in.
• Inference time — even a model trained with perfect privacy still takes live input and produces live output. An attacker can craft a malicious prompt (prompt injection, jailbreaks) to extract context-window data or trigger forbidden behaviour, and the model can leak sensitive information in its responses. None of that is touched by how the model was trained.

The key takeaway: training-time privacy ≠ inference-time security. They are different problems needing different defenses, which is why a model can be "privately trained" and still dangerous in production.

Tip: Think of it as a safe (training) versus a security guard (inference). A well-built safe doesn't help if you hand the guard the keys whenever someone phrases the request cleverly.