Given three characters A, B, C with probabilities $P(A) = 0.5$, $P(B) = 0.25$, $P(C) = 0.25$, what is the entropy and redundancy?

$H(X) = 1.5$ bits, with $H_0 = \log_2(3) \approx 1.585$ bits, giving a redundancy of about 5.4%.

Step-by-step:

$$H(X) = -\left[0.5 \cdot \log_2(0.5) + 0.25 \cdot \log_2(0.25) + 0.25 \cdot \log_2(0.25)\right]$$

$$= -\left[0.5 \cdot (-1) + 0.25 \cdot (-2) + 0.25 \cdot (-2)\right]$$

$$= -\left[-0.5 - 0.5 - 0.5\right] = 1.5 \text{ bits}$$

Maximum entropy:

$$H_0(X) = \log_2(3) \approx 1.585 \text{ bits}$$

Redundancy:

$$R(X) = 1.585 - 1.5 = 0.085 \text{ bits}$$

$$r(X) = \frac{0.085}{1.585} \approx 5.4\%$$

The non-uniform distribution (A is twice as likely) introduces some redundancy, but not much since there are only 3 characters.