Some of the most consequential characters in Unicode are the ones you can’t see. They break searches, logins and spreadsheets, cause baffling bugs in code, and (increasingly) carry hidden instructions to AI systems.
The main families
Zero-width characters
- Zero-width space (U+200B): an invisible break opportunity. Common in text copied from websites and messaging apps.
- Zero-width joiner (U+200D) and non-joiner (U+200C): control how neighbouring characters join. The joiner builds emoji sequences (👨👩👧); the non-joiner is required for correct spelling in Persian and several Indic languages.
- Word joiner (U+2060) and the byte order mark (U+FEFF), which often sneaks into the start of files.
Spaces that aren’t spaces
The no-break space (U+00A0), narrow no-break space (U+202F), en and em spaces, thin and hair spaces, and the ideographic space (U+3000) all look like a normal space but are different characters. A spreadsheet formula, a password check or a search for “New York” fails if one of them is in the way.
Bidirectional controls
Characters such as the right-to-left override (U+202E) change the display order of text. They are needed for some mixed Arabic/Hebrew and Latin text, and they can be abused. In 2021 researchers published “Trojan Source”, showing that bidi controls in source code comments and strings can make code look different to a reviewer than to the compiler. File names have been disguised the same way: inserting a right-to-left override before “fdp.exe” makes a program display as “…exe.pdf”.
Tag characters and “ASCII smuggling”
The block U+E0000–U+E007F contains invisible copies of every ASCII character. They were designed for language tags and today mainly appear in the subdivision flags of England, Scotland and Wales. Security researchers have shown that text written in tag characters is invisible to people but readable by many AI models, so instructions can be hidden inside an innocent-looking paragraph, email or web page (a technique called ASCII smuggling, used in prompt-injection attacks).
Finding them
You can’t find invisible characters by looking. Use a tool that lists code points (a Unicode inspector, a hex editor, or an invisible-character detector that shows each one as a visible label and decodes any hidden tag text).
Removing them safely
Blindly stripping every invisible character causes damage:
- removing joiners and variation selectors breaks emoji;
- removing the zero-width non-joiner misspells Persian and Hindi text;
- removing tag characters inside 🏴 turns Scotland’s flag into a plain black flag.
A careful cleaner keeps these where they belong, converts special spaces to normal spaces, and removes the rest.
Prevention
- Normalize input on forms and in databases (NFC, and NFKC for identifiers).
- Reject or flag bidi controls in source code; many editors and Git hosts now warn about them.
- Sanitize text before it reaches an AI system, especially text from emails and web pages.
Sources
- Unicode Standard, chapter 23 (Special Areas and Format Characters).
- Boucher, N., & Anderson, R. (2021). Trojan Source: Invisible Vulnerabilities. University of Cambridge.
- Unicode Technical Standard #39, Unicode Security Mechanisms.