🔁 HTML to Markdown - Every Tag Traced, Every Refusal Recorded

What went in and what came out

Paste markup, get markdown. The hard part is not the conversion, it is the honesty: markdown cannot spell most of HTML, so a converter either invents something or throws words away, and a page that does either without saying so has cost you an afternoon somewhere between the paste and the commit.

This one keeps three lists and runs one check against your own browser.

• Every element, in the order it was read - what each tag became and the sentence that explains the choice. A <div> that only held other elements disappears on purpose; a <span> with a class on it does not, because the class was the reason it existed.
• Every character the output had to be protected with - the backslashes, and the reason each one is there. & in prose is fine; & in front of copy; is a character reference, so it gets a backslash and a line in this table.
• What was declined - the entries that are not refusals to help but statements that markdown has no construct, or that your markup needed the parser to finish a sentence the author left open.

Below them, the differential: twenty small documents and your own pasted markup, each parsed twice

• once by the tokeniser and the tree builder, once by DOMParser, the same engine that is parsing the tab you are reading this in. Where the two trees agree, the row says so. Where they do not, you are looking at a real disagreement instead of a claim that there are none.

Why the parser is written out in the open

A markdown file is a plain text document with conventions, and HTML is a document with a tree. Going from the second to the first means deciding, for every node, whether markdown has a spelling for it - and then making sure the spelling you wrote is read back as the tree you meant.

The reading half is a tokeniser and a tree builder, both taken from the HTML parsing rules rather than from anybody’s library. The parts that a fragment converter needs are short and strange enough to be worth naming:

• A tag that is never closed still ends at the file. <p>left open gives you a paragraph; <p>one<p>two gives you two, because a second paragraph start tag closes the first one. That rule, and the nine others like it for list items, table cells and headings, is why the second table is not empty.
• A slash at the end of a tag means nothing for most elements. <div/>after leaves the div open and puts “after” inside it. Only void elements - br, img, hr, input and friends - may write themselves closed.
• Formatting tags nest, up to three identical copies. The standard’s own clause is called Noah’s Ark: a fourth <b> while three identical <b> elements are open takes the earliest one out of the list of active formatting tags. That is the list an end tag searches, not the tree - the element it names stays open and keeps the words written after it, so <b>1<b>2<b>3<b>4</b>5</b>6</b>7</b> is four deep and its last 7 sits inside the outermost b. Reading the clause as “close the earliest tag” moves that word outside the run of emphasis it was typed into, and the row in the differential is those two shapes side by side.
• A script body is data. References inside it are not decoded, a < inside it does not open a tag, and its content is not words - so the script’s bytes never come out as a paragraph.
• Character references have edges. &ltt; is <t; and not &ltt; or <;; € is a euro sign, not a control character, because the browsers still apply the Windows-1252 table to the numbers below 32; and inside an attribute, &copy=2 stays exactly as typed while ©=2 becomes ©=2. Each of those is a probe row.
• A table always has a body, and a row always has cells. <table><td>x</table> spells neither a tbody nor a tr, and a browser creates both before it puts the x anywhere. The parser creates them too and marks each one in the node table as a tag it opened for the markup, so that an insertion never shows up as a tag you forgot to close. Building the tree the way the reader will rebuild it is the difference between a converter and a text substituter.
• The names of the references are read from your browser; the rules for them are not. What § stands for is a table of some 2200 entries that browsers ship and revise, so each name is asked of your browser once, the answer kept, and the ones that cannot be found reported. When a semicolon is required, what a name without one may match, and how the same text behaves inside an attribute are the parts written out above - those are rules, and a page that borrowed them from a library would stop being checkable.

What is deliberately not here: the full tree construction algorithm with its foster parenting (text between table rows that the parser relocates), the frameset phases, and template content. Those are the parts of the standard that move your words to a different place in the tree than you wrote them. The tree builder does not move them, which means for markup that triggers the rule the differential will print a disagreement - and that is the honest place to put it, in front of you, rather than in a test file.

Creating a section is not the same as relocating a word, and the two are kept apart: the body and the row opened for a cell-less table go where your markup already put the cell, and nothing you wrote changes hands.

Why emphasis takes the delimiter it takes

foo<em>bar</em>baz is one of the two sentences a markdown converter gets wrong, and the fix is a rule you can read once and keep. An underscore only opens emphasis when it is not sitting between two word characters, so _bar_ inside a word is a literal underscore and the emphasis is gone:

foo_bar_baz        →  foo_bar_baz
foo*bar*baz        →  foo<em>bar</em>baz

An asterisk has no such restriction, so wherever the pair touches a word the writer uses *. The same arithmetic moves a stray space out from under the mark - <em> done </em> becomes *done* - because a delimiter with a space on its inside flanks nothing and opens nothing. And when the same mark nests inside itself, the inner one takes the other character of the pair, which is why you will see _foo *bar* baz_ in the output rather than _foo _bar_ baz_.

There is a third case, and it has no delimiter. a<em>b<em>c</em>d</em>e asks for emphasis inside emphasis of the same kind, and swapping the inner one to the other character of the pair only works where that character can flank - not between two word characters, which is exactly where the nested one usually sits. Writing four asterisks and hoping is the usual answer, and it reads back as two empty ones; the inner tag goes down as markup instead:

a<em>b<em>c</em>d</em>e   →   a*b<em>c</em>d*e

That is the same decision the element policy already makes for a <span> with a class on it, and it puts one line in the refusal table saying which element was given up on. If you see nested-same-mark or mark-unflankable in that table, the markdown beside it is the honest shape, not a mistake.

If your document is full of snake_case identifiers, read the escape table: every underscore that was protected is a place where the plain character would have eaten a word.

Lists, which is where converters fall down

Two decisions carry a list, and both are visible in the tables here.

Loose or tight. A list whose items are separated by blank lines keeps them; one whose items are bare lines does not. In HTML the difference is whether each item holds a paragraph, so that is what the converter looks for. Get it wrong the other way and the nested list below an item stops belonging to it.

Indentation. Each level writes its own items at column zero and lets the level above push them right, so an indent is counted exactly once. A converter that indents on the way down and on the way back up produces the staircase of four, twelve, twenty spaces that a lot of pasted markdown shows - and the reader then sees a code block instead of a list.

Two neighbouring lists are a third problem: markdown has no end-of-list marker, so - a then - b becomes one list whatever the markup said. An empty comment between them closes the first one in any real parser, and that is what gets written, with a line in the escape table saying so.

The three choices

Markup markdown has no spelling for - keep puts the tag into the output as it stands, which is the right answer for a <div class="note"> whose class matters and the wrong one for a page full of tracked changes. escape turns the tag into visible text - useful exactly once, when you want to show somebody markup. drop takes the tag and keeps the words, which is what you want when the markup came out of a word processor and every other element in it is noise.

Links - inline [words](address) sits where it was; reference [words][label] puts every address at the foot of the file, which is the reading you want for a document that cites the same page eleven times. The labels are built from the link’s own words, and a collision is renamed rather than merged, because two labels meaning one address and two addresses meaning one label are both bugs.

Bullet character - only - and *, since a bullet and a space is a list item, and a document whose first bullet is + is a document whose first line is about to stop being one.

Two ways the output will not match your editor

First, whitespace. HTML collapses every run of spaces, tabs and newlines inside text into a single space when it renders, so the same collapse is applied to the words as they are written: a line break in the middle of a sentence is not a line break in the markdown. Only a <br> is treated as a break the author asked for, and it comes out as a backslash at the end of the line.

Second, tables. A pipe table has one header row and no colspan, so a table whose cells span columns or whose alignment differs per column cannot be written as one. Which table was refused, and why, is said rather than emitting a shape that reads back as something else.

What you paste is read here and thrown away

The tokeniser, the tree builder and the writer run in the tab you opened. What you paste is read by a tree built in this tab and thrown away when you close it: the conversion uploads nothing, sends no request, and hands no work to a server. That is also why the differential works at all - it is your browser’s own parser, in your browser, being asked the same question twice.

It is asked in a document made for the purpose and never shown, and that choice is part of the promise rather than an implementation detail. Feeding your markup to an element of the open document would parse it the same twenty ways - every probe in the table agrees between the two - but an image in it would begin loading, and a markup you pasted to be converted has no business sending a request. The throwaway document has no way to reach a server, and it keeps the comment nodes and the script bodies the comparison is about.

Related

If you are going the other way, the markdown to HTML page on this site carries the same four tables pointed the other direction, and reading a document through both of them is a faster way to find what your markup really says than either one alone.