What is the information content (entropy) of a single character $x_i$, and how is it defined?

The information content of a character $x_i$ is $H(x_i) = -\log_2(P(x_i))$ bits — rarer characters carry more information.

* Rarer events carry more bits: a fair coin is 1 bit, one-of-eight is 3 bits, a certain event is 0. *

Shannon's definition:

$$H(x_i) = -\log_2(P(x_i)) = \log_2\left(\frac{1}{P(x_i)}\right)$$

Intuition: Information measures "surprise." If something is very likely ($P \approx 1$), learning it happened gives you almost no information ($H \approx 0$ bits). If something is very unlikely ($P \approx 0$), learning it happened is very informative ($H$ is large).

Examples:

• Fair coin ($P = 0.5$): $H = -\log_2(0.5) = 1$ bit
• One of 8 equally likely characters ($P = 0.125$): $H = -\log_2(0.125) = 3$ bits
• Certain event ($P = 1$): $H = -\log_2(1) = 0$ bits — no surprise, no information

Go deeper:

• Information content (self-information) — the -log2 P(x) definition in depth.
• Entropy (information theory) — how per-symbol information averages into entropy.