In my last post, I talked about simple substitution ciphers used in the ancient world, such as the Caesar shift. While such ciphers are trivial to break today, back then nobody knew how to break them! Thus they were secure.
Cryptanalysis, the science of decoding enciphered messages without the key, did not emerge until the 9th century AD. It required, as a prerequisite, a sophisticated understanding of mathematics, statistics, and linguistics. It was the Islamic civilization which first acquired the necessary knowledge, and which invented cryptanalysis.
The key concept is knowing that some letters appear more frequently in text than others. In Arabic, the letters “a” and “l” are the most common. See the attached graphic for the distribution of letters in the English language. A cipher can be broken by counting the number of times each letter appears in the enciphered text and mapping the most frequently occurring cipher letters to the most common letters in the source language. Some trial and error is often needed, but through educated guessing, the cipher can be solved. This technique is called frequency analysis.
At the time the Arab scholars invented frequency analysis, Europe was still mired in the Dark Ages. Cryptanalysis did not emerge in Europe until the 16th century, and it’s not known whether it was introduced from the Arab world or independently discovered. But once it arrived, substitution ciphers would never be secure in Europe again.