What is Shannon's definition of information, and why does it use the logarithm base 2?

Shannon defined information as $H(x_i) = -\log_2(P(x_i))$ — base 2 gives the result in bits, the fundamental unit of digital information.

Why logarithm?

• Information should be additive: if two independent events both occur, the total information is the sum
• Probabilities are multiplicative: $P(A \cap B) = P(A) \cdot P(B)$ for independent events
• The logarithm converts multiplication to addition: $\log(a \cdot b) = \log(a) + \log(b)$

Why base 2?

• Base 2 gives units in bits (binary digits)
• Base $e$ would give units in nats (natural units)
• Base 10 would give units in hartleys
• Bits are natural for digital systems and computation

Why the negative sign?

• Probabilities are between 0 and 1
• $\log_2$ of a number $< 1$ is negative
• The minus sign makes information content positive: $-\log_2(0.25) = 2$ bits

Claude Shannon published this in his landmark 1948 paper "A Mathematical Theory of Communication" — essentially founding the field of information theory.

Go deeper:

• A Mathematical Theory of Communication — Shannon's 1948 founding paper.
• Information content — the additivity argument for the logarithm.