What Counts as an Invisible Character?
Unicode contains many characters that take up no visible space or look like a blank. They are legitimate in the right place and a nuisance everywhere else. The table groups the ones this tool handles.
| Group | Examples | Usual source | What the tool does |
|---|---|---|---|
| Zero-width and format | U+200B zero-width space, U+2060 word joiner, U+FEFF byte order mark, U+00AD soft hyphen | Web pages, editors, PDFs, copy and paste | Removes them |
| Joiners | U+200C ZWNJ, U+200D ZWJ | Persian, Urdu and Indic-script text; emoji sequences | Safe mode keeps them where they belong and removes them elsewhere |
| Direction marks | U+200E, U+200F, U+061C, U+202A to U+202E, U+2066 to U+2069 | Mixed left-to-right and right-to-left text | Marks stay in right-to-left text under Safe mode; embeddings and isolates are removed |
| Fillers and blanks | U+3164, U+FFA0, U+115F, U+1160, U+17B4, U+17B5, U+2800 | Used to fake blank names or text | Removes them (the Braille blank stays inside real Braille text) |
| Invisible operators and format controls | U+2061 to U+2064, U+206A to U+206F, U+FFF9 to U+FFFC | Copied maths, legacy text | Removes them |
| Variation selectors and tags | U+FE00 to U+FE0F, U+E0100 to U+E01EF, U+E0000 to U+E007F | Emoji and CJK variants; also abused to hide data | Keeps legitimate emoji and flag uses under Safe mode, removes the rest |
| Unusual spaces | U+00A0, U+2002 to U+200A, U+202F, U+205F, U+3000 | Typesetting, CJK text | Replaces them with a normal space (the ideographic space stays beside CJK text under Safe mode) |
| Control characters | U+0000 to U+001F except tab and line breaks, U+007F to U+009F | OCR, terminals, binary data | Removes them (U+0085 becomes a line break) |
Why Invisible Characters Cause Problems
- Search and matching. A word with a hidden character inside it does not match the same word typed normally, so find, filters and exact-match checks miss it.
- URLs, slugs and redirects. A hidden character in a slug creates a different URL from the one you meant, which breaks links, redirects and canonical matching.
- Titles, meta descriptions and keywords. Text that looks fine in an editor can still fail exact-match checks or look odd in snippets.
- Spreadsheets, CSV and databases. Two keys that look identical are different strings, so lookups and joins fail.
- Code and configuration. Stray characters cause unexpected-token errors, and direction controls can disguise what code actually says during review.
- Usernames and passwords. A pasted invisible character is a common reason a login that looks right is rejected.
How Safe Mode Decides What to Keep
Some invisible characters do real work. The zero-width non-joiner shapes Persian and Urdu words, the zero-width joiner builds Indic conjuncts and emoji such as family and profession sequences, and tag characters spell out flags such as England and Scotland. Safe mode looks at the characters around each one. It keeps a joiner next to Arabic or Indic-script letters or inside an emoji sequence, keeps direction marks in right-to-left text, keeps emoji variation selectors that follow an emoji, and keeps tag characters only inside a valid flag sequence. The same characters in the middle of an English word are removed. After a Scan, the report lists what was kept and why it was kept.
Remove Invisible Characters in Code
If you would rather do it in your own pipeline, these snippets cover the common cases. The first two lines leave Persian joiners and emoji alone, while the strict version removes every default-ignorable character, including those.
// JavaScript: the usual zero-width characters only
const cleaned = text.replace(/[\u200B\u2060\uFEFF\u00AD]/g, '');
// JavaScript: everything Unicode treats as default-ignorable (also removes ZWJ, ZWNJ, variation selectors)
const strict = text.replace(/\p{Default_Ignorable_Code_Point}/gu, '');# Python
import re
cleaned = re.sub(r'[\u200b\u2060\ufeff\u00ad]', '', text)