🔣 HTML Entity Escape and Unescape — & < Numeric References and the Browser's Lenient Rules

Two ways text breaks when it goes into markup

Text reaches a web page in two bad ways. One is pasting <b> or a user’s <script> and having it render as markup. The other is running something that is already escaped through a second pass, so your own page shows a literal &. Both need a converter that knows how a browser actually reads HTML, not one that just replaces five characters.

So there are two directions here. A escapes: ordinary text becomes something safe to write into markup. B unescapes: entity references (&, 中, 😀, &nbsp) become text again. B is written against the browser’s own reading: a named table, the legacy names that never needed a semicolon, the Windows-1252 numeric range, the surrogate rules. Every decoding rule was compared case by case against what the browser’s parser produces.

A · Which characters must move

• In element content (<p>here</p>): three things — & → &, < → <, > → >. A bare > is technically legal almost everywhere, but sequences like --> and ]]> do break things, so it always gets escaped.
• In an attribute value (title="here"): add " → " and ' → '. With only a double-quoted attribute you would only need ", but the tool does not know which quote you will type, so “attribute mode” (on by default) escapes both and the result is safe inside either. Switch it off and you get the content mode, with quotes untouched.

Escaping is one-way, not idempotent: & goes in and & comes out, so text that has already been escaped once will show a literal & in the browser. Nothing here can tell whether your input is already HTML, and it does not guess.

With “numeric references for non-ASCII” on, every code point above 0x7F becomes &#decimal;, one reference per code point: the emoji 😀 is two UTF-16 units inside JavaScript but comes out as a single 😀, never as half a character. This option is for feeding systems that only accept ASCII; on an ordinary UTF-8 page plain Chinese text is easier to read, and the source of the text you are reading does exactly that.

B · How references are read

• &name;: the tool looks for the longest name that is followed by a semicolon. The table has 187 entries — the five basic ones in both spellings, the Latin-1 accented letters in upper and lower case, punctuation and typography, currency and units, Greek letters, arrows and maths. A name that is not in the table and ends with ; stays put and is counted under “unrecognised references”.
• Legacy names without the semicolon: HTML 4’s ISO 8859-1 set never needed the closing ;, and browsers still honour that. With the switch on, &copy → © and &nbsp → no-break space, using the longest matching prefix: &notitice becomes “¬itice” rather than an error, because the longest legacy name inside it is not. Names that arrived later — hellip, mdash, euro, trade, apos — are not in that list and must be written in full. Turn the switch off and anything without a semicolon is left exactly as it was.
• Numeric references: &# plus decimal digits, or &#x plus hex; the value is read as a code point. The semicolon can be omitted (&#65 → A) and the note counts how many were.
• A lone &: in AT&T the & is not followed by a valid name, so it is not a reference, it stays, and it is not counted as unrecognised. AT&T is a reference and decodes.

Numeric references in a few ranges must not simply become “the character with that number”. This is what browsers really do:

Because control characters really do come back, the result box can end up holding invisible text: copying it gives you those exact characters, and a <textarea> displays a \r as a line break. That is the box, not the decoder.

When to check the output yourself

• A is not an XSS defence on its own. It guarantees “this text appears as text”. Content going inside <script>, <style> or a URL needs different treatment — inside a script only </script ends it, and < there is not a less-than sign. Let your template engine do the context-aware escaping.
• Newlines do not survive escaping by themselves. HTML collapses source newlines into whitespace. Use <br> (which means allowing markup, the opposite of escaping) or white-space: pre-line.
• What B produces is not automatically safe to insert. <b> decodes to <b>, and from that moment it is markup. Validation between decoding and insertion is your job.
• The name table is curated, not complete. The specification lists 2231 named references (all those maths monsters, and even the capital Greek letters such as Α). The 187 here are the ones you meet in text; the rest fall into “unrecognised” and stay as written. Use numeric references for anything exotic.
• Do not treat the lenient rules as a target. &nbsp works for historical reasons; new code should always write the semicolon, and &amp is a hard error in XHTML. The switch exists because the text you pasted was probably written that way.

Questions

Q: What is the difference between µ and μ? They are not the same character. µ is U+00B5 MICRO SIGN, the unit prefix from ISO 8859-1; μ is U+03BC, the Greek letter. They look identical and differ in code point, which matters for search and comparison. Both decode to their own code point and are never merged.

Q: Why not just hand the text to the browser with innerHTML? Because that feeds the whole string to an HTML parser, where an <img onerror> runs immediately. The scanning script only walks characters: no eval, no innerHTML, and nothing you paste is placed in a request, so it does not leave this device. It also means one set of rules gives the same answer in every browser, rather than whatever each parser happens to do with broken input.

Q: Why does ’ become a curly quote rather than control character 146? Because U+0092 would be invisible and useless, the spec routes the whole 0x80-0x9F block through Windows-1252, where position 146 is ’. Anyone who treats “the number is the Unicode code point” in that range gets a string of invisible characters instead of punctuation — which is exactly why the table above exists.

Q: How much can it handle? 4000 characters per box; beyond that only the first 4000 are processed and the note says so. Both boxes recalculate 60 ms after you stop typing and they are independent, so you can copy A’s output straight into B to check that a decode gives back what you started with.