What is the entropy $H(X)$ of a random variable (information source), and what is the maximum entropy $H_0(X)$?

$H(X)$ is the weighted average information content across all characters; $H_0(X) = \log_2(n)$ is the maximum entropy achieved when all characters are equally likely.

Entropy of an information source $X$ with $n$ characters:

$$H(X) = -\sum_{i=1}^{n} P(x_i) \cdot \log_2(P(x_i))$$

This is the expected (average) number of bits per character.

Maximum entropy (decision content):

$$H_0(X) = \log_2(n)$$

$H_0$ is achieved when and only when all characters are equally likely: $P(x_i) = \frac{1}{n}$ for all $i$.

Key insight: $H(X) \leq H_0(X)$ always. Any non-uniform distribution has less entropy than the uniform distribution over the same alphabet. This is why natural language text (with non-uniform letter frequencies) can be compressed — it has redundancy.

Go deeper:

• Entropy (information theory) — the weighted-average definition and its maximum.
• A Mathematical Theory of Communication — Shannon's founding 1948 paper.