🔤 Text Encoding Repair - What These Bytes Actually Say, and What Can Still Be Given Back

Why one sentence has many byte spellings

A character is a number, and an encoding is a rule for writing that number as bytes. The two characters 中文 are U+4E2D U+6587, and UTF-8 spends six bytes on them - E4 B8 AD E6 96 87 - while GBK spends four, D6 D0 CE C4. Both files are correct. Neither file says which rule was used, unless somebody left a mark at the front, and even the mark only names one family of rules out of nine.

That is the whole reason a byte reader is worth writing, and the reason none of them can promise an answer. Reading a file’s bytes through the wrong rule is not a crash - it is a perfectly ordinary string of the wrong characters, and the only way to find the right rule is to try each one and look at what comes back. So the table below shows all nine readings side by side, each in two columns: what your own browser’s decoder reports, and what the byte ranges say when the run is walked without the decoder. Where those two disagree, the disagreement is the interesting part.

What makes a UTF-8 sequence illegal, and why nobody else’s rules can

UTF-8 is the strictest of the nine, and it is strict for reasons that are all about safety rather than about saving bytes. A lead byte promises how many bytes follow, and the ranges are tight:

Eight of the nine readings can refuse a byte for a reason like these - the four multi-byte families each have their own lead and trail ranges, and the two UTF-16 readings refuse an odd byte count and an unpaired surrogate - while Windows-1252 cannot, because a 256-cell table has a name for every one of its cells and is never asked to explain a byte. That is why both columns are printed: an answer for every byte is not evidence that any of the answers is right.

The hand-written walker refuses the same sequences a browser does - C0 AF, E0 80 80, ED A0 80, F0 80 80 80, F4 90 80 80, F5 80 80 80 all come back as errors, and a run that simply stops in the middle of a sequence is refused too, with the byte it stopped on named.

Three kinds of damage, and only two of them can be undone

The bytes are fine and only the viewer was wrong. This is the common case and the easy one: the file on disk still holds what it was written with, and something opened it with the wrong default. Point the picker at the file, or paste it, and force the reading you want. Nothing has been lost.

The wrong reading was saved out as text. Here is the arithmetic of it, measured rather than remembered: 中文 in UTF-8 is E4 B8 AD E6 96 87. Read those six bytes as GBK and you get three characters - U+6D93 U+E15F U+6783, which is 涓 followed by a code point from the private-use area your font has no drawing for, followed by 枃. Save that as UTF-8 and the file now holds nine bytes, E6 B6 93 EE 85 9F E6 9E 83. Undoing it means encoding those three characters back into GBK’s two-byte pairs to get the six bytes again, and that needs the other half of the GBK table - all 23,940 cells of it. No such table is carried here, on purpose, which is why the paste box refuses and tells you which characters stopped it. Give it the file instead and the bytes are already there, so it can walk them for free.

The data is gone. A decoder that cannot use a byte writes the replacement mark, U+FFFD, which in UTF-8 is EF BF BD. Read two of those marks as GBK and the pairs come out as 锟斤拷 - the three characters that mean somebody gave up on your bytes before you ever got the file. The mark is not a spelling of the original, it is a record that the original was thrown away, and many different originals reach the same mark. Nothing can put it back, and the honest answer names the damage and stops.

One accident really is reversible: a byte run that was read through cp1252 (or its label latin1, which browsers treat as the same table). Every character in that misreading sits under U+0100, so each one names exactly one byte, and the run can be rebuilt without a table at all - 中æ becomes E4 B8 AD E6 96 87 again, and then re-reads as 中文. The five bytes cp1252 leaves undefined (81, 8D, 8F, 90, 9D) are passed through unchanged by browsers, so they rebuild too. The moment a character above U+00FF shows up, the rebuild is over and the note says so with positions.

The mark at the front cuts both ways

EF BB BF is three bytes that mean “this is UTF-8”, written by the programs that were afraid you would not guess. They are not part of the text, and what happens to them is the most common complaint a byte reader gets:

• A UTF-8 decoder removes it before you ever see it. The decoder is asked not to, the mark is shown in the table, and the walker strips it, so you can see whether the file has one.
• Excel’s CSV import wants it. Without those three bytes Excel reads the file through whatever code page your Windows was set to, and the Chinese in it arrives as 涓; with them it arrives correct.
• JSON.parse rejects it - a document that begins with the mark is a syntax error, measured here rather than remembered.
• A hash you publish changes if the mark is in the file, because the hash is over bytes and the mark is bytes.
• Read through GBK, the three marks are the characters 锘 and a stray; read through cp1252 they are . Both are what “锘縤d,name” and “id,name” complaints are.

The other marks name themselves: FF FE and FE FF are UTF-16’s, and a UTF-16 file that opens with one of them is the one case where certainty is possible, because the mark names the byte order and the reading then has to take every byte. FF FE 00 00 is the awkward one: it is UTF-32’s little-endian mark, and it is also UTF-16LE’s mark followed by a NUL character, which is why four bytes are looked at before anything is committed. UTF-32 it will not read: no browser’s TextDecoder implements it, and inventing a four-byte-at-a-time guess in a web page is not the same as shipping the codec.

Detection reads shape, not a corpus

Encoding detection in the literature is a corpus business: statistical models over how often a language uses which characters, and a big table of them. No corpus and no pair tables are carried here, so only shapes can be looked at, and the note says when shapes are not enough:

• A reading that hands back control characters or replacement marks is nearly always wrong, so those cost it.
• A reading whose characters scatter across several script blocks is wrong in a way real text is not: misreading CJK pairs through the wrong family lands you in Hangul, or in the private-use area, or in three of those at once. Actual text in one language stays inside one or two blocks. The ranking is mostly this one observation.
• A reading that puts most of its characters in one CJK, kana or Hangul family gets a bonus, and that is a guess about shape, not about your language.
• A run of pure ASCII has no evidence in it at all. All seven byte-wide readings return the same characters, so that is the sentence that gets said, instead of a detection that was never available; the only thing in such a file that points anywhere is a NUL byte, because a NUL in every other position is what Western text looks like once it is written two bytes per character.
• When several readings take the whole run and score the same, the table says so, prints what each one gives, and lets you choose. Nothing is averaged.

If you force a reading from the list, the ranking is not consulted at all - you have already decided - and the note says which readings refused a byte outright, because that is still useful information about the file.

Why the output can only ever be UTF-8

Saving back in the original encoding is the first thing everyone asks for, and a browser cannot do it. TextEncoder has one encoding, and it is UTF-8; hand it {encoding: "gbk"} and it accepts the option, reports its own name as utf-8 and writes UTF-8 bytes anyway. Decoding is nine tables; encoding is one.

So the picker has three honest settings and no fourth: write the bytes exactly as they arrived, which is what you want when only the viewer was wrong and you are just moving the file; write the text as UTF-8 without a mark, which is what JSON, hashes, git and every Unix tool want; or write it with the mark, which is what Excel wants. Anything else - GBK, Big5, EUC-KR, Shift_JIS output - needs the table that is refused here, and belongs in iconv, Python’s codecs, .NET’s Encoding.GetEncoding, or Excel’s own “save as” dialog.

Questions

Q: Does my file get uploaded anywhere? No. The bytes are read into this tab by the browser’s own file reader and walked by loops in this tab’s script. The file is never offered to a server, and no server ever sees the text - which matters more than usual here, because the files people bring are exports of orders, rosters and patient lists.

Q: Why does it show me nine readings instead of an answer? Because bytes alone usually cannot settle it. A file that does not name its encoding has exactly two pieces of real evidence - a mark at the front, and the readings that refuse a byte - and when neither is decisive, showing all nine is the honest answer.

Q: It opens as garbage in one program and correctly in another. Which do I believe? Neither: the file did not change, only the default each program picks when the file refuses to name itself. Pick the reading here until the text looks like your language, then save it with the mark in front so the next viewer cannot guess wrong.

Q: Can you undo 锟斤拷? No, and nothing will be pretended otherwise. Those characters are what the replacement mark EF BF BD looks like when it is read through GBK, which means a decoder already gave up on the original bytes and wrote a placeholder over them. Many different originals produce that same mark, so there is nothing left to reverse.

Q: I pasted 涓枃-style garbage and it says the bytes cannot be rebuilt. Why? Because each of those characters is a GBK pair, and turning it back into two bytes means looking it up in the 23,940-cell GBK table. The ranges of that table are carried, which is enough to walk a file’s bytes, but not the pairs themselves, which is what the reverse direction needs. Attach the file instead of pasting its garbage and the bytes are already there.

Q: It says the answer is open, or lists three readings tied. What do I do? Read them against each other in the table and keep whichever one spells your language: only you know whether the line is supposed to be Chinese, Korean or a Windows accident. The ranking is a guess about character shapes, and the ranking is there to be overruled, not obeyed.

Q: How large a file can this handle? Up to 2 MB of bytes. A file bigger than that is not a repair job for a textarea; a real editor will hold it, and the encoding question is settled in the first few hundred bytes anyway.