🧹 Text Dedupe and Sort by Line — Remove Duplicates, Empty Lines and Sort Naturally

The cleanup jobs hands get wrong

Text lists need the same four chores over and over: a column copied out of a spreadsheet has half its rows twice, a field list contains both user_id and User_ID, a log has three thousand ERROR lines that really say eleven different things, and sorting item2 / item10 / item20 the usual way gives item10, item2, item20.

Each of those has a rule you could state out loud, which is exactly why doing it by hand goes wrong. So the tool is split into two switchable halves — A cleans and merges (strip whitespace, drop empty lines, ignore case, keep the first or the last copy) and B orders (lexicographic, natural, by length, reversed). The four counters show where the lines went, and the line underneath names which values were merged, how many times each appeared and on which input lines — the one thing a dedupe tool usually refuses to tell you.

When two lines count as one

Comparison is on the whole cleaned line, character by character, with no normalisation of any kind:

• With whitespace stripping on (the default), apple and apple are one item. With it off they are two.
• With ignore-case on, the comparison uses the lowercased line, so Apple, apple and APPLE collapse into one. Which spelling survives depends on whether you keep the first or the last copy.
• Inner spaces are always content: new york and new york (two spaces) stay separate, so does a half-width versus a full-width space.
• Line endings do not matter on the way in: \r\n, \n and a lone \r (very old Mac files) all split a line. Output uses \n only.
• The final newline of the input does not add a phantom empty line, so a list that ends with a line break reads as exactly the number of items you typed.

Case folding is the generic Unicode one, not locale-aware, so German ß and SS are not merged (they should not be), and a Turkish İ lowercases into an i plus a combining mark. Ignore-case does nothing for Chinese, Japanese or Korean — those scripts have no upper or lower form, so nothing can merge that way.

The three orders

“Sort the result” is the master switch; leave it off and the tool only dedupes, keeping everything in the order you pasted it. With it on:

• Plain (lexicographic): compare by UTF-16 code unit, smallest first. This is deliberately blunt and predictable — spaces and punctuation come before letters (-a before aa), every capital before every lowercase (Zoo before apple, because Z is 90 and a is 97), and CJK characters sit at their own code points, which is not pinyin order.
• Natural order: a run of digits is compared as one piece, so item2 precedes item10 and v9 precedes v10. The run is compared by “strip leading zeroes, then length, then characters” and is never converted to a number, so id99999999999999999998 and id99999999999999999999 stay distinct instead of rounding together. The version with more leading zeroes sorts first (item007 before item7). Only ASCII 0-9 counts as a digit — a full-width ３ is treated as an ordinary character.
• By character count: shortest line first; lines of equal length then fall through to one of the two orders above. Digits get no special treatment here, so item2 (5) precedes item10 (7). Length is counted in UTF-16 units, so one emoji is length 2 and a line made only of an emoji ties with ab.

“Reverse” negates every comparison step, which for a list with no two identical lines is the same as flipping the sorted result. With sorting switched off it acts alone and simply reverses the deduped list — handy for logs where the newest line is at the bottom.

Order of the steps

The stages run in a fixed sequence, which is what makes the result reproducible:

1. Split on line breaks.
2. Strip the ends of each line (if asked).
3. Drop empty lines (if asked) — “empty” means nothing left after step 2, so with stripping off a line of spaces is not empty and survives.
4. Merge duplicates, keeping the first or the last copy.
5. Sort (if asked).
6. Join with \n into the result box.

When to check the result yourself

• Semantic duplicates stay: 1,000 and 1000, Ⅲ and III, a name with a zero-width space in it — different lines by any of these rules. Normalise first if you need them merged.
• Which copy you keep matters: the default keeps the first. For append-only lists (logs, exports where later rows win) tick “keep the last copy”. The surviving line stays at its own position, it is not moved to the end of the list.
• Sorting Chinese text: code-point order correlates with nothing a reader expects (no pinyin, no stroke count), so sorting is only useful there as a way of grouping identical prefixes.
• Very large lists: 40000 characters, a few thousand short lines. Past that only the first 40000 are read and the note says so; hundreds of thousands of lines belong in a command-line tool, not a browser tab.
• Invisible characters: trim removes the standard whitespace set (spaces, tabs, \n, \r, no-break space U+00A0, ideographic space U+3000, BOM U+FEFF) but not zero-width spaces (U+200B) or directional marks (U+200E/U+200F). If two visibly identical lines refuse to merge, that is usually why.

Questions

Q: Why does “lines in” count blank lines? It counts the lines that were split out, not the ones with content; the separate “empty lines removed” counter says how many were dropped. With “drop empty lines” off that counter shows “—”, meaning not applicable.

Q: Why is the duplicate list capped at eight entries? Because a list with four hundred different repeated values would bury the eight you care about. The entries shown are the most frequent ones (ties broken by position in the input), and the tail of the line states how many more were not listed.

Q: Can it sort Chinese or Japanese text the way a dictionary does? No. For Chinese that needs a table of thousands of polyphonic characters plus a decision on how to read each one in context — dictionary-scale work, and guessing wrong would make the output less predictable, not more. What you get is code-point order.

Q: Does the pasted text get uploaded? Splitting, comparing and sorting are loops inside the script: no eval, and no copy of your list is sent to anything. What you paste does not leave this device.