ADR-0057: Content Record Is the ProseMirror Document Tree; Markdown Becomes a Derived Projection¶
- Status: accepted
- Amended by: ADR-0058 — §5's dual-format coexistence is retired pre-production (PM_DOC-only); §§1–4/6–7 unchanged.
- Date: 2026-07-10
- Deciders: Nicolas Burri, after the structural discussion of the round-trip whack-a-mole ("I am open to discuss deep cutting changes… We will need to have full traceability but from that point on, everything else is open for discussion.")
- Supersedes: the storage half of ADR-0045 (markdown text as the source of record) FOR NEW VERSIONS; ADR-0045's rendering-policy principles (sanitization, no raw HTML, no external images, doc:/attachment: schemes) carry over unchanged.
- Retires: ADR-0055 §3/§4 (fidelity gate as adoption criterion, per-document entry guard incl. all amendments) and the ADR-0056 §3 canon-fencing machinery — for tree-format versions there is no round trip to guard. ADR-0055 §1/§2 (Milkdown engine, mode-switch UX) and ADR-0056 §1/§2 (Crepe chrome) stand.
- Relates to: ADR-0001 (blob storage — mechanism unchanged), ADR-0044 (search), ADR-0007 (validation evidence).
Context — why the text record failed us¶
The editing arc (ADR-0055/0056) stored free-form markdown TEXT as the record and required the
visual editor's parse→serialize to reproduce it byte-exactly (entry guard). Live experience
produced an unbounded stream of byte-divergence classes (fidelity report D1–D8: trailing
newlines, trailing spaces, table padding, bare-URL ambiguity, <br /> emission, and finally the
editor's own trailing-paragraph artifact making saves non-idempotent), plus pipeline-replica
drift (the guard's probe diverged from the live pipeline within 48 hours of a fix). Root cause:
markdown text admits infinitely many byte representations of the same document while the editor
holds exactly one tree — byte equality at that boundary adjudicates human idiosyncrasy forever,
and EVERY future editor feature (math is already requested) re-enters that business.
Ecosystem precedent: Confluence stores ADF, a ProseMirror-style JSON tree, and renders/diffs from it. Session probes verified the mechanics on our stack: headless DOMSerializer rendering (raw HTML escaped; policy layer still mandatory for schemes), plugin-diff position-mapped version deltas, deterministic tree→markdown serialization.
Decision¶
- The authoritative content of a document version is the ProseMirror document JSON (the Milkdown/Crepe schema), stored through the existing SHA-256 blob mechanism (ADR-0001) with a dedicated media type. Frozen versions remain immutable blobs exactly as today.
- Markdown becomes a derived, deterministic, canonical PROJECTION — computed at save time
by ONE shared projection function (the single pipeline; fixed-point property
project(import(project(t))) == project(t)is a standing test over the existing corpus), stored alongside the tree per version. It serves: human-readable record copy, text diffs (canonical-vs-canonical — cleaner than today), the AI/MCP content surface, and exit-strategy insurance against schema lock-in. - Plain-text search extraction is computed SERVER-SIDE by walking the JSON tree (structural text-node concatenation needs no ProseMirror library) — the search index can never drift from the record and never depends on the client-computed projection.
- The projection is client-computed (the projection function is remark/ProseMirror JavaScript). Trust model: it is part of the authenticated author's save, audit-attributed like the content itself; determinism is enforced by the shared function + corpus tests, and an offline verification tool (Node) can recompute projections against stored trees as an integrity audit. Accepted trade documented here deliberately.
- No bulk rewrite of history. Existing versions keep their markdown-text blobs forever
(
content format v1); version rows carry a format marker. The reading view dispatches on format (markdown-it path for v1, tree renderer for v2). A document migrates when it is next REVISED: the new draft imports (parses) the old markdown once — the honest, visible, audit-attributed conversion — and is tree-format from then on. Rollback at any point = stop creating v2 versions; nothing is destroyed. - Editing modes over the tree: visual mode loads/saves the tree natively (no parse, no serialize, NO GUARD — the friction class ceases to exist, including the unsupported-construct problem: whatever the tree holds is exactly what renders). Source mode becomes an editor over the PROJECTION: switching/saving imports text→tree (a total function — textual idiosyncrasies normalize on import BY DESIGN, visibly, not by ambush).
- Reading view renders from the tree (DOMSerializer on the shared schema) behind the UNCHANGED policy layer (DOMPurify final authority, doc:-link rewriting, attachment-scheme resolution, external-image refusal, mermaid/template-vars handling) — editor and reader share one grammar; the markdown-it path remains solely for v1 legacy versions.
Known limitations (owned consciously)¶
- Projection of markdown-poor features (Nicolas's flagged concern, e.g. future LaTeX math):
where standard syntax exists (
$...$) the projection uses it; where markdown cannot express a node, the projection DEGRADES DOCUMENTED (per-feature note in the projection module) rather than blocking the feature. The tree remains lossless; projection fidelity is a documentation duty per feature, no longer a correctness gate. - Raw record readability: the stored tree is JSON, not prose. Mitigation: the stored canonical projection is the human-readable copy, generated at save, immutable per version.
- Schema evolution is now a managed surface: adding node types is backward-compatible; renaming/removing requires an explicit, tested migration of tree blobs (each such change gets its own ADR).
Consequences¶
- The fidelity corpus + harnesses convert into the projection's determinism/fixed-point tests;
the guard/convert UI, D8 machinery and
roundtripThroughPipelineare retired with format v2. - Search (ADR-0044) gains a server-side extraction column; AI/MCP consumers get canonical markdown — both improve over free-form authored bytes.
- Realization plan, phases and gates: tree-record-migration.md.