Incident: A guessed encoding turned text into Chinese

Module 3 · Incident file · About 30 minutes

What happened

In the early 2000s a strange trick went round the internet. Open Notepad, the plain text editor that comes with Windows, and type the sentence Bush hid the facts. Save the file, close it, and open it again. Instead of the sentence, Notepad showed a short row of Chinese characters. People passed it on as a hidden message. It was nothing of the kind.

The cause was an encoding guess. A plain text file holds only bytes; it does not record which encoding turned the characters into those bytes. When Notepad opened a file, it had to work out the encoding for itself. To do that it used a Windows function called IsTextUnicode, which looks at the bytes and guesses whether they are UTF-16, an encoding that uses two bytes for each ordinary character. For some short plain-text files, such as this sentence, the guess came out as UTF-16. Notepad then read every pair of English letters as one character, and the pairs happened to be Chinese ones.

Nothing on the disk was damaged. The bytes the typing put there were still the right ones. What broke was the reading: the program chose an encoding the writer never used, and nothing on screen said it had guessed. The case is still told as the classic example of an encoding guess gone wrong (widely reported, early 2000s; Microsoft's documentation for the IsTextUnicode function describes it as a test that determines whether a buffer is likely to contain Unicode text).