🔎 Character Codes - Every Byte, Every Cluster, Every Rule Written Down

What “one character” means, three ways

A sentence you paste is three different things at once, and the three counts never agree except by luck.

• UTF-16 units - what String.length answers with, because JavaScript strings are sequences of 16-bit units. Everything outside the first 65,536 code points costs two of them, so a smiley is 2 to length and 1 to your eye.
• Code points - the number Unicode actually assigned. 👨‍👩‍👧 is five of these: three pictures and two joiners.
• Clusters - what a browser draws as one glyph. The same five code points are one cluster, which is why the family emoji takes up one cell in a text field and one arrow key press.

Switch the row table to units and a pair shows up as the two rows it really is. A unit that is half of a pair has no character of its own, so its row prints a dash where the glyph and the bytes would go - only the pair has those, and inventing a code point for one half would be making something up. A half with no partner next to it is a different thing, and gets its own row, saying which half it is and what an encoder does with it.

All three are read off the same string, in the same pass, and the arithmetic is printed. The “one row per” selector picks which of the three you look at; the counts across the top are always all three, so a surprise between them is visible without switching anything.

The four tables

• Everything you pasted, one row at a time - the character, its code point, which plane and block it sits in, its UTF-8 bytes in the notation you chose, what the carried ranges can say about it, and what your browser’s own Unicode tables say about it. The last two columns disagree on purpose: one is arithmetic over the sixty-odd mark ranges and eighty-odd block ranges written out above, the other is a 2,000-page data file that came with your browser.
• How the text is cut into clusters - every cluster the walk produced, the code points inside it, and the rule that kept them together. This is the table to read when the count in the header does not match the one your browser gives below.
• The same text in every notation - UTF-8 bytes, UTF-16 big- and little-endian, HTML decimal and hexadecimal character references, percent-encoding, a CSS identifier, and a JavaScript string literal. Each row is written out and then handed to the engine in your browser that reads that notation, and the “round trip” column says whether the same text came back. Where it did not, the row is the interesting one: those are the places where an encoding has a rule rather than a formula.
• Declined to answer - four standing refusals that apply to every string, plus whatever your paste specifically walked into. The standing ones are the character names (a 34,000-line data file, so the block is given instead), full canonical decomposition (only the Hangul arithmetic is carried), the bidi, width and line-break classes (three more tables, decided by a terminal or a browser, not by the range tables), and named HTML references like © - those live on /tools/html-entities/, which reads the name list out of your browser as it goes.

The rules the cluster walk uses, all of them

Stated once, so you can check them against the table above:

Give every code point one of a short list of names - a line ending, a control, a mark or a variation selector or a skin tone or a tag, a joiner, a regional indicator, one of the five Hangul classes, or plain. Then put a boundary between two neighbours unless one of these says not to: a CR followed by an LF is one ending; a mark, a variation selector, a skin tone or a tag sticks to whatever precedes it; so does a joiner, and a joiner takes the character after it as well, but only when both sides of it are in the picture list; the Hangul classes build a syllable; regional indicators pair off two by two. Everything else starts a new cluster, including a lone control.

That walk only ever looks at the character immediately before the one it is deciding about, which is the source of its two blind spots. A browser that follows the published segmentation rules will keep a Devanagari or Malayalam consonant pair - two consonants with a vowel-sign between them - as one cluster, because that rule needs to know which script each character belongs to, and script is a table that is not carried. And a handful of marks that are written in front of a number - the Arabic number signs, a few in the Tibetan and Khmer ranges - are one cluster with their digit for a browser and two for the walk. Paste either and the differential row says so.

Reading the differential

The last table runs the same string through the carried arithmetic and through your browser’s own implementations - TextEncoder for UTF-8, Intl.Segmenter for clusters, String.prototype.normalize for Hangul, encodeURIComponent, CSS.escape, the property escapes like \p{M}, TextDecoder for the two UTF-16 byte orders, and an HTML parser for the character references - and prints both answers.

Rows where the browser has no such API print “not checked” instead of vanishing, because a table that changes shape depending on which engine you happen to be running is a table nobody can read twice. Three more things leave a row unchecked, and all three are the language rather than the ranges carried. A string with a surrogate half in it has no percent-encoding, because encodeURIComponent throws on it instead of answering. A numeric character reference for zero, for a surrogate half, or for the numbers 80-9F is not read back as the character it names - HTML answers U+FFFD for the first two and the old Windows-1252 character for that window - so the row prints which of those the parser did to your text and the refusal table counts them. And the same half that an encoder will not percent-encode is a byte pattern a decoder will not hand back: the UTF-16 rows write your string’s code units, halves included, because that is what those bytes are, and TextDecoder answers one U+FFFD for each half instead of returning it. That row says how many of them your own decoder did that to. A difference the arithmetic cannot account for like this is still a marked difference.

A row that prints different is not a bug in the carried ranges or in your browser; it is one of three things, and the prose above covers each of them. Either the carried ranges are narrower than the published property (a mark from a block that is not listed, a digit outside 0030-0039, of which there are about six hundred), or the segmentation rule needs a table (the two cases in the previous section), or your browser’s data file is a different Unicode version from the ranges written out here. That last one matters for the picture list in particular: it is a transcription of a published list of code points, not a rule, so it ages, and the row “the transcribed picture list” is exactly that check - if your browser’s tables and the carried list put different code points on the picture side, the row prints the count and names the first few.

One difference is settled and does not belong to any of those three, so it is said out loud instead of being left in the table: encodeURIComponent leaves !, ', (, ) and * as they came, because RFC 3986 keeps those five unreserved for backward compatibility, while the percent encoder writes the byte for every character outside the letters, digits, -, ., _ and ~. Paste one of them and the row shows the two spellings, and a refusal names how many of them your text had.

Limits, and why they are that size

4,000 UTF-16 units of input, 300 rows in each table, 96 characters in each cell before it is cut with a note. The input limit is there because a string is read character by character and a row is printed for each one; past a few thousand, the tables are a file to open rather than a page to read, and the counts above them are computed on the whole input anyway. The Save button hands you that file.

All of it runs on the machine that typed the text. No request is made with your text, no font is fetched, no parser is called on a server: the pasted string never leaves this device. The one place a string goes into a parser is the HTML read-back, and there only numeric character references are written into a detached document made for the purpose - never markup you pasted - so nothing in it can load or render anything.