ADR-0092: Orphan-Blob Garbage Collection With a Quarantine Bin¶
- Status: accepted (Nicolas 2026-07-22 — GAP-row decision "2a", with the bin refinement: "potentially using a 'bin' to soft delete orphaned blobs and only to delete them permanently after a certain time, e.g. 30 days in the bin")
- Date: 2026-07-22
- Deciders: Nicolas Burri, design by Claude. Origin: the 2026-07-22 GAP investigation found
that REQ-DOC-013's GC job does not exist, while
DisposalService(ADR-0042) explicitly relies on it — its post-commit blob deletion logs failures and continues on the promise that "orphan garbage collection sweeps any leftover blob (ADR-0025)". Orphans currently leak forever. - Relates to: ADR-0001 (content-addressed store), ADR-0025 (disposal/erasure mechanics — the GC assumption this ADR discharges), ADR-0042 (disposal execution), ADR-0037 (scheduler/sweep election pattern).
- Requirement: REQ-DOC-013 — "A garbage-collection job shall remove blobs no longer referenced by any version; referenced blobs shall never be deleted."
Decision¶
- Two-phase deletion — quarantine, then purge. A blob is never removed from disk the first
time it is seen unreferenced. Phase 1 (quarantine): a periodic sweep records each orphan in a
blob_quarantineregistry (scope, hash, quarantined_at). Phase 2 (purge): a later sweep pass permanently deletes a quarantined blob only after it has sat in the bin for the configured quarantine period (default 30 days, configurable) and is re-verified unreferenced at purge time. - The bin is a registry, not a file move. Quarantined blob bytes stay physically in place in
the content store; only the registry row marks them. This keeps the content-addressed store's
invariants intact: an idempotent
putof the same content (re-reference through dedup) simply works,getnever breaks, and rescue is a registry-row delete. Any sweep pass that finds a quarantined hash referenced again removes it from the bin (rescue) — the safety re-check at purge time makes even a missed rescue harmless. - Reference truth is the database, enumerated explicitly. A blob is live when its hash is referenced by any non-disposed row of a blob-referencing column. The implementation enumerates these columns in one place (the GC's reference query), and an architecture test asserts the schema's set of hash-carrying blob-reference columns equals that enumeration — adding a new blob-referencing column then fails the test until the GC inventory is extended (the schema-extension lifecycle trap, made structural).
- Safety invariant, structurally enforced: referenced blobs are never deleted. Both phases run their reference check and the registry mutation in one transaction; physical file deletion happens post-commit (the ADR-0042 pattern), is idempotent, and a failed file deletion simply leaves an orphan that the next sweep re-quarantines — the mechanism is self-healing in both directions.
- Scheduling and observability: a nightly scheduler tick (advisory-lock election, the
ADR-0037
TaskSweeppattern — every instance ticks, one sweeps). Each pass logs per-scope counts (quarantined, rescued, purged); a purge that actually deletes blobs writes an audit event with the counts, so the destructive act is visible in the trail without flooding it on idle nights.
Consequences¶
- Additive migration (
blob_quarantine), alist(scope)enumeration onContentStore(the filesystem implementation walks its scope directory), the sweep service + scheduler, config for period and quarantine duration, tests incl. the never-delete-referenced invariant, the quarantine round-trip (orphan → bin → rescue on re-reference; orphan → bin → purge after expiry) and the column-inventory architecture test. - Discharges the
DisposalServicepost-commit failure path's standing assumption; REQ-DOC-013 flips from GAP to verified via the new tests. - No user-facing surface for the MVP: the bin is an operational safety net, not a restore feature; rescue is automatic (re-reference), not manual.
Amendment 2026-07-25 — the quarantine row is the deletion MUTEX (review finding A3)¶
- Status: amendment implemented 2026-07-25 (Opus 5 review round, finding A3 — HIGH, data-loss class); RATIFIED by Nicolas 2026-07-26 ("the ADR looks good"). It refines the mechanism of §4 in order to keep §4's own safety invariant ("referenced blobs are never deleted"). The decision itself — two-phase bin, registry not file move, DB reference truth, nightly election — is unchanged.
What was wrong. §4 said both phases run their reference check and the registry mutation in one
transaction, with physical deletion post-commit (the ADR-0042 pattern). The re-check was a snapshot
reuse, not a re-read: one live-set read served quarantine, rescue and purge, so "re-verified
unreferenced at purge time" was tautological. Worse, post-commit deletion cannot be made safe by
re-checking harder. put is content-addressed and DEDUPS (byte-identical content writes nothing), so a
concurrent upload can commit a live reference to a hash between the check and the file deletion — the
reference survives, the bytes do not, including under a frozen content part of a RELEASED record. The
same shape existed in DisposalService's post-commit deletion.
The amendment.
- The quarantine row is a mutex, and both sides take it. A blob's file is deleted only by a
transaction that first CLAIMED its bin row (a
DELETE … RETURNING), and the deletion happens INSIDE that transaction, while the claim's row lock is held. A guarded write (BlobGcGuard.put) deletes the same row as its RESCUE, on the writer's own transaction. For a given (scope, hash) exactly one side wins: the claimer (the writer then blocks, finds no row, and re-materializes its bytes — the SELF-HEAL), or the writer (the row is gone, the hash is never claimed, the file is never touched). - The purge re-reads liveness against the claim — a fresh statement snapshot under READ COMMITTED, not the pass's opening snapshot. A claimed hash found live again is rescued instead of purged.
- Every production blob write goes through the guard (
BlobGcGuard.put): freeze body/tree/manifest, attachment upload, derive attachment copy, training evidence. A bareContentStore.putis a write outside the protocol. - Disposal marks before it destroys. The severed-and-unreferenced hashes enter the bin INSIDE the disposal transaction and are destroyed immediately after it commits, through the same claim+purge primitive. Erasure stays prompt (no 30-day wait) and stays outside the disposal transaction, so a storage error still cannot roll back a committed tombstone; a failure now leaves the blobs quarantined with their purge clock already started instead of as unnoticed orphans.
Consequence for §4. "Physical file deletion happens post-commit" no longer holds for the GC purge: post-commit deletion drops the mutex before the destructive act. The ADR-0042/0043 rationale for post-commit deletion — never roll back a committed, audited tombstone because of a transient storage error — does not apply to the GC, which has no tombstone to protect: a purge that rolls back simply leaves the blob quarantined for the next sweep. Self-healing in both directions is unchanged; a failed file delete leaves an orphan the next sweep re-quarantines.
Residual, accepted. The claim is a blocking DELETE, so a deadlock is theoretically possible when one
transaction writes two blobs whose expired bin rows a purge is claiming in the opposite order. PostgreSQL
detects it and aborts one transaction; nothing is deleted unsafely and the sweep retries on the next tick.
SELECT … FOR UPDATE SKIP LOCKED would remove even that, but PostgreSQL requires the UPDATE privilege for
row locking (verified empirically against postgres:18-alpine), and V065 deliberately withheld UPDATE on
blob_quarantine to keep bin rows immutable. Not worth a grant-widening migration for a race this narrow.
Still on the old pattern (registered follow-up). Three lower-traffic deletion paths still delete
post-commit without the mutex and carry the same (smaller) window: attachment replace/delete
(AttachmentService), reject/cancel orphan reclamation (DocumentService.deleteOrphanBlobsBestEffort,
called from LifecycleService), and training-evidence delete (TrainingPlanService). They now have the
primitive to adopt — mark in the transaction, claim + purgeClaimed after it — and adopting it is
mechanical; it was left out of the A3 fix to keep that change reviewable.
Known limitation (recorded 2026-08-21, ruled 2026-08-23 — amendment deferred past 1.0)¶
The quarantine mutex covers two orderings; a third exists that it does not: a writer whose put,
rescue and existence check ALL run before the deleter's mark, and whose reference commits after
the purge, still loses its bytes — purgeClaimed's liveness re-read cannot see an uncommitted
INSERT. The exposure requires a write and a delete of the same (scope, hash) interleaving within
one transaction window; it is registered as RISK-005's residual. The fix is a shared lock on
(scope, hash) taken by put as well — a contract change every blob caller inherits, which is
why Nicolas ruled it a deliberate post-1.0 amendment rather than a rushed one. Until then this
paragraph is the honest ceiling: the mutex makes destruction-vs-rescue safe, not write-vs-delete
races within a single window.