What is the entropy of the German language (per character), and what does the Markov entropy tell us?

German text has $H(X) \approx 4.089$ bits/char based on single-letter frequencies, but the Markov entropy (accounting for letter dependencies) drops to about 1.6 bits/char.

Single-letter analysis (memoryless model):

• Alphabet: 27 characters (26 letters + ß)
• $H_0(X) = \log_2(27) \approx 4.755$ bits/char (maximum)
• $H(X) \approx 4.089$ bits/char (actual, due to non-uniform letter frequencies)
• $R(X) = 4.755 - 4.089 = 0.666$ bits/char redundancy

With letter dependencies (Markov model):

• Accounting for digram frequencies (e.g., "qu" is common, "qx" is not), the entropy drops further
• Markov entropy of German: $\approx 1.6$ bits/char
• This means 1 million German characters contain only ~1.6 million bits of actual information

Why this matters for cryptography:

• High redundancy ($r \approx 66\%$) makes classical ciphers vulnerable to frequency analysis
• An attacker can exploit predictable letter patterns to break simple substitution ciphers
• Compression before encryption is a good idea: it removes redundancy, making the ciphertext more uniform and harder to attack
• This redundancy is also why human language text can be compressed so effectively (ZIP, gzip)

Tip: The gap between single-letter entropy (4.089) and Markov entropy (1.6) shows how much information is contained in letter dependencies — nearly 2.5 extra bits per character come from knowing which letters tend to follow which.

Go deeper:

• Frequency analysis — how language redundancy enables cipher-breaking.
• Redundancy (information theory) — the single-letter vs. Markov entropy gap.