Skip to content
LQMS
LQMS Server Deployment (single Linux host, public HTTPS)
Initializing search
nburri/lqms
LQMS
nburri/lqms
Home
Evaluate LQMS
Evaluate LQMS
LQMS Known Limits
LQMS Security Posture
Architecture
Architecture
1. Introduction and Goals
2. Architecture Constraints
3. Context and Scope
4. Solution Strategy
5. Building Block View
6. Runtime View
7. Deployment View
8. Crosscutting Concepts
9. Architecture Decisions
10. Quality Requirements
11. Risks and Technical Debt
12. Glossary
The arrival bundle format
Decisions (ADRs)
Decisions (ADRs)
ADR-0001: Content-Addressed Document Storage with PostgreSQL as Source of Truth
ADR-0002: Single Instance with Scope Hierarchy and Copy-on-Derive
ADR-0003: Fixed Life Cycle States with Configurable Review Processes
ADR-0004: Global Catalogs for Roles and Document Types with Per-Scope Activation
ADR-0005: Modular Monolith
ADR-0006: Three-Layer Search with In-Database Index and Pluggable LLM
ADR-0007: Angular Frontend with Server-Side-Only Enforcement
ADR-0008: Flyway Migrations and jOOQ Codegen from the Migrated Schema
ADR-0009: RLS Enforcement via a Session-Scoped Authorized Scope Set
ADR-0010: Self-Managed QMS via an Internal Mandator
ADR-0011: Authentication via OIDC SSO with Server-Side (BFF) Sessions
ADR-0012: Permission-Based Authorization and System Bootstrap
ADR-0013: User Identity Lifecycle
ADR-0014: Electronic Signatures (Part 11 / Annex 11-Capable)
ADR-0015: Document Effectivity, Periodic Review & Validity
ADR-0016: Data Retention, Disposal & Privacy (FADP / GDPR)
ADR-0017: Multi-Part Version Content
ADR-0018: Non-Functional Targets (Capacity, Availability, Security)
ADR-0019: Notifications & Tasks
ADR-0020: Internationalization (Content Language, Time Zones, Formatting)
ADR-0021: General Typed Document Relationships
ADR-0022: Read-Acknowledgement & Questionnaires
ADR-0023: Migration / Import — Current Released State
ADR-0024: Draft / In-Review Cancellation
ADR-0025: Disposal & Erasure Mechanics (Storage, Index, Backups)
ADR-0026: RLS Runtime Safety — Connection-Pool Scope Isolation
ADR-0027: RLS Boundary — Content-Plane vs Control-Plane
ADR-0028: Stateless Encrypted Token-State Cookie (no server-side session store)
ADR-0029: Document Content Model — Draft-Freeze, Content-Table RLS, Append-Only Audit
ADR-0030: Review & Life-Cycle Realization Model
ADR-0031: Released-base readability enforcement & copy-on-derive mechanics
ADR-0032: Frontend realization — Quinoa, Angular Material, i18n & E2E
ADR-0033: CSRF control for the BFF session — SameSite + JSON content-type
ADR-0034: Notification realization — in-process domain events + system-context fan-out
ADR-0035: Fan-out contract as shipped — gate semantics, count disclosure, audit protocol
ADR-0036: Repository layer — services carry rules, repositories carry jOOQ
ADR-0037: Task scheduler — in-process tick, advisory-lock election, SQL sweep
ADR-0038: Periodic-review realization — derived due date, policy-gated confirmation
ADR-0039: Sweep & periodic-review contract as shipped — grants, guards, addressing
ADR-0040: Disposal-due surfacing — closure-anchored retention, scope-level legal hold
ADR-0041: Email channel & notification i18n — notification-as-outbox, structured messages
ADR-0042: Disposal execution — document tombstoning, single-actor hard-gated destruction
ADR-0043: Disposal blob deletion is post-commit, GC-backstopped
ADR-0044: Search FTS baseline — event-indexed, RLS-scoped search_index
ADR-0045: Image Attachments as Content Parts (Rich-Content Slice R2)
ADR-0046: Four-Eyes — Submitter Excluded from Approval
ADR-0047: Four-Eyes — Submitter-Only Exclusion
ADR-0048: Document Organization — Folder Tree per Scope
ADR-0049: Content Model Round 2 — Type Categories, Template Instantiation, Trace Relations, Completeness Rules
ADR-0050: Live Frontend — SSE State Push, CodeMirror 6, Draft Presence
ADR-0051: In-Text Document Links (doc: Scheme)
ADR-0052: Addressees & Read-Acknowledgement (REQ-TRN Realization)
ADR-0053: Addressees Per Document + Process Owner (amends ADR-0052 K-1)
ADR-0054: Process Owner Is a Role (amends ADR-0053 P-3)
ADR-0055: WYSIWYG Editing Mode over Markdown (editor step R4)
ADR-0056: Visual-Editor Chrome via Milkdown Crepe (gated adoption)
ADR-0057: Content Record Is the ProseMirror Document Tree; Markdown Becomes a Derived Projection
ADR-0058: PM_DOC Is the Only Content Format (pre-production cut; amends ADR-0057 §5)
ADR-0059: Editor Engine Switches to TipTap (spike-gated; supersedes the Milkdown/Crepe choice)
ADR-0060: Backup & Restore — Scripted Ordered Backup with Automated Restore Rehearsal
ADR-0061: CSV — Generated Validation Package per Release
ADR-0062: SBOM — CycloneDX over the Production Layers, Validated Against the Artifact
ADR-0063: User Identity Binds on Subject Only (Issuer Becomes Provenance)
ADR-0064: Per-Document Workflow Role Bindings (Author / Reviewer / Releaser), Enforced
ADR-0065: Diagrams — Excalidraw as the Manual Layer, Mermaid Stays for Text-First/UML
ADR-0066: WITHDRAW — the Author Side May Reclaim a Version from Review
ADR-0067: Review Comments — Anchored, Immutable, Resolvable
ADR-0068: Confluence/Jira QMS Import — Staged, Read-Only, Repeatable
ADR-0069: Arbitrary File Attachments — Download-Only Confinement
ADR-0070: Training Questionnaires — Versioned, Auto-Scored, Release-Gated
ADR-0071: Training Plans — Document Obligations + External Trainings & Certifications
ADR-0072: Diagrams — Self-Hosted draw.io Replaces Excalidraw
ADR-0073: Per-Scope QMS Configuration Surface
ADR-0074: PDF Export of Released Documents — Server-Side via Apache FOP over the Canonical Projection
ADR-0075: Source History Import — Continuity Without Forged Audit
ADR-0076: Periodic-review status visibility (completing MVP gap #2)
ADR-0077: Requirement Items — Keyed Blocks Inside Controlled Documents
ADR-0078: Generalized Trace Items (Traceability Model — Stage A)
ADR-0079: Risk & Risk-Control Items (Traceability Model — Stage B)
ADR-0080: Traceability Workspace — Hub, Sub-Pages, and Role-Gated Views
ADR-0081: Draft-Only Trace-Structure Changes, Server-Enforced
ADR-0082: Table-Shaped Risk Authoring over Risk Blocks
ADR-0083: The TD Dossier — a Versioned, Cross-Scope, Regulatory-Structured Deliverable
ADR-0084: Validation & Regulatory Axes (Traceability Model — Stage C)
ADR-0085: Relation-Layer Consolidation — One Trace Vocabulary, One Surface
ADR-0086: Requirement Traceability Chains — End-to-End Path View + Flat Path Export
ADR-0087: The Scope Traceability Report — One Derived Overview Artifact, Never a Controlled Document
ADR-0088: Verification Evidence Classes — Planned vs. Verified, Derived at the Read Layer
ADR-0089: Recurring Obligations — Periodic Tasks With Record Evidence, Not Calendar Series
ADR-0090: Release & Versioning Policy — Deliberate Releases, One Auditor-Facing Number
ADR-0091: Release Modes — Single-Person Record Release With a Structural Four-Eyes Floor
ADR-0092: Orphan-Blob Garbage Collection With a Quarantine Bin
ADR-0093: One Scope Context — the Toolbar Switcher Is the Only Scope Selector
ADR-0094: Calendar-Anchored Obligation Schedules (Clamped, Never Too Late)
ADR-0095: Key-Rule Governance — Free Until First Use, Then Frozen
ADR-0096: Transition Permissions — Always Visible, Deny Unless Granted
ADR-0097: A Governed Provisioning Surface — MANAGE_SCOPES
ADR-0098: Org Chart × Role System — Marriage at the Evidence Level, Never in Authorization
ADR-0099: The User Manual is an In-App Topic Reference, Not a Document
ADR-0100: LQMS Is the Living Quality Management System
ADR-0101: The Executable Validation Journey (PQ Evidence That Cannot Drift)
ADR-0102: Typed Template Variables (text | date | number — and Deliberately No Enum)
ADR-0103: npm audit Joins the Verification Gate as a Second Standing Input
ADR-0104: The Issued Dossier Is a Persisted Artifact of Record — and Clones to a New Draft
ADR-0105: Navigation Information Architecture — Group by Moment-of-Use
ADR-0106: The Client Reflection Follows Assignments, Not Only Content Reach
ADR-0107: The Audit Append Gate Follows Assignment Reach, Not Content Reach
ADR-0108: Mandator Membership — Invite-Accept, with Pre-Assigned Responsibilities
ADR-0109: Organization Founding Is Its Own Process
ADR-0110: The Audit Trail Is Not a Read Model
ADR-0111: The Imported-Person Model, and Project Portability's First Doctrines
ADR-0112: The Element-Kind Catalog — Semantics in the Model, Not Only in the Code
ADR-0113: Project Arrival Import — Records Arrive as They Are, People Arrive as They Were
ADR-0114: The Operating Model Arrives — Roles as Functions, Names as Words, People as This Instance's Decision
ADR-0115: Collaborative Draft Editing — Two Carets, One Truth, Honest Credit
ADR-0116: The System Maintenance Surface — the App Backs Itself Up, the Host Keeps One Dumb Job
ADR-0117: Role Documentation — a Link, a Finding, Per-Source Recognizers; Substitution Rejected
ADR-0118: The Offline QMS Copy — the Repo Carries the Audit Pack, Not a Second Renderer
ADR-0119: Module-Internal Package Structure — Capabilities First, Layers Second, the Root Is the Surface
ADR-0120: The Shared Kernel — the Domain-Exception Vocabulary Leaves the Authorization Module
ADR-0121: The Project Administrator Role — Per-Scope Administration With Read Access, Seeded in the Base Catalog
ADR-0122: Organization-Level Administration — the Cascade Stays, Becomes Visible, and Founds Projects
ADR-0123: Draft Save-Points — Every Save Restorable, Kept for the Life of the Version
ADR-0124: The Birth Path Completes Without Borrowing Content Clothes
ADR-0125: The Export Twin — Export Emits an Arrival Bundle
ADR-0126: Project Dissolution — the Un-Birth of an Unused Scope
ADR-0127: The Base-QMS Layers — One Library, Derived Adaptations, Local Records
ADR-0128: The GLOBAL Cascade Carries Administration, Never Content
ADR-0129: Role Identity from a GLOBAL Assignment Acts in GLOBAL Alone
ADR-0130: Break-Glass — the Project Administrator's Escalation Is a First-Class, Witnessed, Self-Expiring Act
ADR-0131: Role Substitution as Declared Adoption Edges — Emergency Cover Is Governed, Own-Name, Timed Staffing
ADR-0132: Dual-Hat Reviewer Satisfaction Is a Declared Policy Choice — the Strict Floor Stays the Default
ADR-NNNN:
</span> </a> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" > <div class="md-nav__link md-nav__container"> <a href="../../requirements/" class="md-nav__link "> <span class="md-ellipsis"> Requirements </span> </a> <label class="md-nav__link " for="__nav_5" id="__nav_5_label" tabindex="0"> <span class="md-nav__icon md-icon"></span> </label> </div> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false"> <label class="md-nav__title" for="__nav_5"> <span class="md-nav__icon md-icon"></span> Requirements </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../../requirements/stakeholder_requirements/" class="md-nav__link"> <span class="md-ellipsis"> Stakeholder Requirements </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/gap-analysis/" class="md-nav__link"> <span class="md-ellipsis"> Requirements Gap Analysis </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_admin/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Administration, Permissions & Onboarding </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_api_ui/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: API & User Interface </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_audit_reporting/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Audit Trail & Regulatory Reporting </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_auth/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Authentication & Session </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_backup/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Backup & Restore </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_catalogs/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Catalogs, Roles & Users </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_data_protection/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Data Protection, Retention & Disposal </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_documents/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Documents, Versions & Storage </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_dossiers/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Technical Documentation Dossiers </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_effectivity/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Effectivity, Periodic Review & Validity </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_gspr/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Regulatory Conformity (GSPR / Essential Principles) </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_i18n/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Internationalization </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_lifecycle/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Life Cycle & Reviews </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_memberships/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Mandator Membership & Invitations </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_migration/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Migration & Import </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_nfr/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Non-Functional </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_notifications/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Notifications & Tasks </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_obligations/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Recurring Obligations </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_relationships/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Document Relationships </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_risk/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Risk Management (ISO 14971 Hazards & Controls) </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_sbom/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Software Bill of Materials & Supply-Chain Assurance </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_search/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Search </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_separation/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Scopes, Mandators & Separation </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_signatures/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Electronic Signatures </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_training/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Read-Acknowledgement & Questionnaires </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_users/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: User Identity & Lifecycle </span> </a> </li> <li class="md-nav__item"> <a href="../../requirements/req_validation/" class="md-nav__link"> <span class="md-ellipsis"> Software Requirements: Design Validation & the Validation Traceability Axis </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_6" > <div class="md-nav__link md-nav__container"> <a href="../../process/" class="md-nav__link "> <span class="md-ellipsis"> Process (IEC 62304) </span> </a> <label class="md-nav__link " for="__nav_6" id="__nav_6_label" tabindex="0"> <span class="md-nav__icon md-icon"></span> </label> </div> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_6_label" aria-expanded="false"> <label class="md-nav__title" for="__nav_6"> <span class="md-nav__icon md-icon"></span> Process (IEC 62304) </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../../process/configuration_management_plan/" class="md-nav__link"> <span class="md-ellipsis"> Configuration Management Plan </span> </a> </li> <li class="md-nav__item"> <a href="../../process/problem_resolution/" class="md-nav__link"> <span class="md-ellipsis"> Problem Resolution Process </span> </a> </li> <li class="md-nav__item"> <a href="../../process/risk_management/" class="md-nav__link"> <span class="md-ellipsis"> Risk Management </span> </a> </li> <li class="md-nav__item"> <a href="../../process/software_development_plan/" class="md-nav__link"> <span class="md-ellipsis"> Software Development Plan </span> </a> </li> <li class="md-nav__item"> <a href="../../process/verification_plan/" class="md-nav__link"> <span class="md-ellipsis"> Verification Plan </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--active md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_7" checked> <label class="md-nav__link" for="__nav_7" id="__nav_7_label" tabindex="0"> <span class="md-ellipsis"> Operations </span> <span class="md-nav__icon md-icon"></span> </label> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_7_label" aria-expanded="true"> <label class="md-nav__title" for="__nav_7"> <span class="md-nav__icon md-icon"></span> Operations </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../genesis-installation/" class="md-nav__link"> <span class="md-ellipsis"> Genesis: Birthing an LQMS Installation with a Base Library </span> </a> </li> <li class="md-nav__item"> <a href="../deployment/" class="md-nav__link"> <span class="md-ellipsis"> LQMS Deployment Guide </span> </a> </li> <li class="md-nav__item md-nav__item--active"> <input class="md-nav__toggle md-toggle" type="checkbox" id="__toc"> <label class="md-nav__link md-nav__link--active" for="__toc"> <span class="md-ellipsis"> LQMS Server Deployment (single Linux host, public HTTPS) </span> <span class="md-nav__icon md-icon"></span> </label> <a href="./" class="md-nav__link md-nav__link--active"> <span class="md-ellipsis"> LQMS Server Deployment (single Linux host, public HTTPS) </span> </a> <nav class="md-nav md-nav--secondary" aria-label="Table of contents"> <label class="md-nav__title" for="__toc"> <span class="md-nav__icon md-icon"></span> Table of contents </label> <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix> <li class="md-nav__item"> <a href="#1-topology" class="md-nav__link"> <span class="md-ellipsis"> 1. Topology </span> </a> <nav class="md-nav" aria-label="1. Topology"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#why-keycloak-sits-on-a-subdomain-not-a-path" class="md-nav__link"> <span class="md-ellipsis"> Why Keycloak sits on a subdomain, not a path </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#2-sizing" class="md-nav__link"> <span class="md-ellipsis"> 2. Sizing </span> </a> </li> <li class="md-nav__item"> <a href="#3-prerequisites" class="md-nav__link"> <span class="md-ellipsis"> 3. Prerequisites </span> </a> <nav class="md-nav" aria-label="3. Prerequisites"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#31-dns--two-records-before-the-first-start" class="md-nav__link"> <span class="md-ellipsis"> 3.1 DNS — two records, before the first start </span> </a> </li> <li class="md-nav__item"> <a href="#32-firewall" class="md-nav__link"> <span class="md-ellipsis"> 3.2 Firewall </span> </a> </li> <li class="md-nav__item"> <a href="#33-unattended-os-updates" class="md-nav__link"> <span class="md-ellipsis"> 3.3 Unattended OS updates </span> </a> </li> <li class="md-nav__item"> <a href="#34-the-deploy-files-on-the-server--delivered-by-scp-not-by-git" class="md-nav__link"> <span class="md-ellipsis"> 3.4 The deploy files on the server — delivered by scp, not by git </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#4-the-image--published-by-ci-pulled-by-the-server" class="md-nav__link"> <span class="md-ellipsis"> 4. The image — published by CI, pulled by the server </span> </a> <nav class="md-nav" aria-label="4. The image — published by CI, pulled by the server"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#41-the-package-is-private--confirm-it-once" class="md-nav__link"> <span class="md-ellipsis"> 4.1 The package is private — confirm it once </span> </a> </li> <li class="md-nav__item"> <a href="#42-the-servers-registry-credential--a-classic-pat-with-readpackages" class="md-nav__link"> <span class="md-ellipsis"> 4.2 The server's registry credential — a classic PAT with read:packages </span> </a> </li> <li class="md-nav__item"> <a href="#43-before-you-cut-the-release-tag-the-deploy-runtime-render-check" class="md-nav__link"> <span class="md-ellipsis"> 4.3 Before you cut the release tag: the deploy-runtime render check </span> </a> </li> <li class="md-nav__item"> <a href="#44-if-you-must-build-it-yourself" class="md-nav__link"> <span class="md-ellipsis"> 4.4 If you must build it yourself </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#5-secrets" class="md-nav__link"> <span class="md-ellipsis"> 5. Secrets </span> </a> <nav class="md-nav" aria-label="5. Secrets"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#51-inventory--what-must-be-regenerated" class="md-nav__link"> <span class="md-ellipsis"> 5.1 Inventory — what must be regenerated </span> </a> </li> <li class="md-nav__item"> <a href="#52-how-the-realm-gets-them-vardefault-substitution" class="md-nav__link"> <span class="md-ellipsis"> 5.2 How the realm gets them: ${VAR:default} substitution </span> </a> </li> <li class="md-nav__item"> <a href="#53-inviting-people-who-have-no-account-yet--mail-and-the-one-credential-that-mints-logins" class="md-nav__link"> <span class="md-ellipsis"> 5.3 Inviting people who have no account yet — mail, and the one credential that mints logins </span> </a> <nav class="md-nav" aria-label="5.3 Inviting people who have no account yet — mail, and the one credential that mints logins"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#existing-installations-the-realm-was-imported-once-so-add-the-client-by-hand" class="md-nav__link"> <span class="md-ellipsis"> Existing installations: the realm was imported once, so add the client by hand </span> </a> </li> <li class="md-nav__item"> <a href="#existing-installations-credential-self-service-register-2026-08-06-27" class="md-nav__link"> <span class="md-ellipsis"> Existing installations: credential self-service (register 2026-08-06 #27) </span> </a> </li> </ul> </nav> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#6-bring-up" class="md-nav__link"> <span class="md-ellipsis"> 6. Bring-up </span> </a> <nav class="md-nav" aria-label="6. Bring-up"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#61-first-content-wizard-birth-or-demo-seed" class="md-nav__link"> <span class="md-ellipsis"> 6.1 First content: wizard birth or demo seed </span> </a> </li> <li class="md-nav__item"> <a href="#62-existing-installations-give-the-tenant-its-real-name" class="md-nav__link"> <span class="md-ellipsis"> 6.2 Existing installations: give the tenant its real name </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#7-do-not-run-the-demo-scripts-against-a-server-stack" class="md-nav__link"> <span class="md-ellipsis"> 7. Do not run the demo scripts against a server stack </span> </a> <nav class="md-nav" aria-label="7. Do not run the demo scripts against a server stack"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#71-base-file-only-scripts-and-the-pulled-image" class="md-nav__link"> <span class="md-ellipsis"> 7.1 Base-file-only scripts and the pulled image </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#8-keycloak-admin-access" class="md-nav__link"> <span class="md-ellipsis"> 8. Keycloak admin access </span> </a> </li> <li class="md-nav__item"> <a href="#8b-importing-a-corpus--the-one-shot-rule-read-before-the-production-import" class="md-nav__link"> <span class="md-ellipsis"> 8b. Importing a corpus — the one-shot rule (read before the production import) </span> </a> </li> <li class="md-nav__item"> <a href="#8c-exporting-a-corpus--the-portability-bundle-adr-0125" class="md-nav__link"> <span class="md-ellipsis"> 8c. Exporting a corpus — the portability bundle (ADR-0125) </span> </a> </li> <li class="md-nav__item"> <a href="#9-backups" class="md-nav__link"> <span class="md-ellipsis"> 9. Backups </span> </a> <nav class="md-nav" aria-label="9. Backups"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#91-the-schedule--what-the-reference-installation-actually-runs" class="md-nav__link"> <span class="md-ellipsis"> 9.1 The schedule — what the reference installation actually runs </span> </a> </li> <li class="md-nav__item"> <a href="#92-the-off-server-copy-is-the-point" class="md-nav__link"> <span class="md-ellipsis"> 9.2 The off-server copy is the point </span> </a> </li> <li class="md-nav__item"> <a href="#93-restore-rehearsal--preferably-not-on-this-box" class="md-nav__link"> <span class="md-ellipsis"> 9.3 Restore rehearsal — preferably not on this box </span> </a> <nav class="md-nav" aria-label="9.3 Restore rehearsal — preferably not on this box"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#what-a-rehearsal-record-proves--and-why-the-pre-2026-08-19-rows-do-not" class="md-nav__link"> <span class="md-ellipsis"> What a rehearsal record proves — and why the pre-2026-08-19 rows do not </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#94-restoring-onto-the-server--the-exact-sequence" class="md-nav__link"> <span class="md-ellipsis"> 9.4 Restoring onto the server — the exact sequence </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#10-rotating-a-secret-after-first-boot" class="md-nav__link"> <span class="md-ellipsis"> 10. Rotating a secret after first boot </span> </a> </li> <li class="md-nav__item"> <a href="#11-keeping-it-current" class="md-nav__link"> <span class="md-ellipsis"> 11. Keeping it current </span> </a> <nav class="md-nav" aria-label="11. Keeping it current"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#111-upgrading-lqms--the-whole-ritual" class="md-nav__link"> <span class="md-ellipsis"> 11.1 Upgrading LQMS — the whole ritual </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#12-first-deploy-checklist" class="md-nav__link"> <span class="md-ellipsis"> 12. First-deploy checklist </span> </a> </li> <li class="md-nav__item"> <a href="#13-troubleshooting" class="md-nav__link"> <span class="md-ellipsis"> 13. Troubleshooting </span> </a> <nav class="md-nav" aria-label="13. Troubleshooting"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#131-issuer-mismatch--the-classic-breakage" class="md-nav__link"> <span class="md-ellipsis"> 13.1 Issuer mismatch — the classic breakage </span> </a> </li> <li class="md-nav__item"> <a href="#132-login-loop--the-session-cookie-is-dropped" class="md-nav__link"> <span class="md-ellipsis"> 13.2 Login loop — the session cookie is dropped </span> </a> </li> <li class="md-nav__item"> <a href="#133-certificates-are-not-issued" class="md-nav__link"> <span class="md-ellipsis"> 13.3 Certificates are not issued </span> </a> </li> <li class="md-nav__item"> <a href="#134-exec-format-error-when-the-app-container-starts" class="md-nav__link"> <span class="md-ellipsis"> 13.4 exec format error when the app container starts </span> </a> </li> <li class="md-nav__item"> <a href="#135-the-app-refuses-to-boot-fan-out-deployment-check" class="md-nav__link"> <span class="md-ellipsis"> 13.5 The app refuses to boot: fan-out deployment check </span> </a> </li> <li class="md-nav__item"> <a href="#136-database-authentication-failure-after-a-restore" class="md-nav__link"> <span class="md-ellipsis"> 13.6 Database authentication failure after a restore </span> </a> </li> <li class="md-nav__item"> <a href="#137-http-431--a-wedged-browser-profile" class="md-nav__link"> <span class="md-ellipsis"> 13.7 HTTP 431 / a wedged browser profile </span> </a> </li> <li class="md-nav__item"> <a href="#138-the-image-will-not-pull" class="md-nav__link"> <span class="md-ellipsis"> 13.8 The image will not pull </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#14-what-this-deployment-deliberately-does-not-do" class="md-nav__link"> <span class="md-ellipsis"> 14. What this deployment deliberately does not do </span> </a> </li> </ul> </nav> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_8" > <div class="md-nav__link md-nav__container"> <a href="../../traceability/" class="md-nav__link "> <span class="md-ellipsis"> Traceability </span> </a> <label class="md-nav__link " for="__nav_8" id="__nav_8_label" tabindex="0"> <span class="md-nav__icon md-icon"></span> </label> </div> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_8_label" aria-expanded="false"> <label class="md-nav__title" for="__nav_8"> <span class="md-nav__icon md-icon"></span> Traceability </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../../traceability/matrix/" class="md-nav__link"> <span class="md-ellipsis"> Requirements Traceability Matrix </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_9" > <div class="md-nav__link md-nav__container"> <a href="../../validation/" class="md-nav__link "> <span class="md-ellipsis"> Validation (tool lane) </span> </a> <label class="md-nav__link " for="__nav_9" id="__nav_9_label" tabindex="0"> <span class="md-nav__icon md-icon"></span> </label> </div> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_9_label" aria-expanded="false"> <label class="md-nav__title" for="__nav_9"> <span class="md-nav__icon md-icon"></span> Validation (tool lane) </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../../validation/templates/ai-authorship-posture/" class="md-nav__link"> <span class="md-ellipsis"> AI-Authorship Posture — How LQMS Is Built, and Why the Validation Evidence Does Not Rest on It (TEMPLATE) </span> </a> </li> <li class="md-nav__item"> <a href="../../validation/templates/intended-use/" class="md-nav__link"> <span class="md-ellipsis"> Intended-Use Specification — LQMS as a QMS / Document-Control Tool (TEMPLATE) </span> </a> </li> <li class="md-nav__item"> <a href="../../validation/templates/tool-risk-assessment/" class="md-nav__link"> <span class="md-ellipsis"> Tool-Risk Assessment — LQMS as a QMS Tool (TEMPLATE) </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_10" > <label class="md-nav__link" for="__nav_10" id="__nav_10_label" tabindex="0"> <span class="md-ellipsis"> Regulatory </span> <span class="md-nav__icon md-icon"></span> </label> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_10_label" aria-expanded="false"> <label class="md-nav__title" for="__nav_10"> <span class="md-nav__icon md-icon"></span> Regulatory </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../../regulatory/mdr-annex-i-skeleton/" class="md-nav__link"> <span class="md-ellipsis"> MDR Annex I — GSPR authoring skeleton (paraphrased) </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item md-nav__item--nested"> <input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_11" > <div class="md-nav__link md-nav__container"> <a href="../../user-guide/" class="md-nav__link "> <span class="md-ellipsis"> User guide </span> </a> <label class="md-nav__link " for="__nav_11" id="__nav_11_label" tabindex="0"> <span class="md-nav__icon md-icon"></span> </label> </div> <nav class="md-nav" data-md-level="1" aria-labelledby="__nav_11_label" aria-expanded="false"> <label class="md-nav__title" for="__nav_11"> <span class="md-nav__icon md-icon"></span> User guide </label> <ul class="md-nav__list" data-md-scrollfix> <li class="md-nav__item"> <a href="../../user-guide/tutorial/" class="md-nav__link"> <span class="md-ellipsis"> Tutorial — from zero to a released document </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/administration/" class="md-nav__link"> <span class="md-ellipsis"> Administering an LQMS Installation </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/diagrams/" class="md-nav__link"> <span class="md-ellipsis"> Diagrams </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/documents-lifecycle/" class="md-nav__link"> <span class="md-ellipsis"> Documents & the Lifecycle </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/exports/" class="md-nav__link"> <span class="md-ellipsis"> PDF Exports & the Audit Pack </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/obligations/" class="md-nav__link"> <span class="md-ellipsis"> Recurring Obligations </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/project-context/" class="md-nav__link"> <span class="md-ellipsis"> The Project Context & App Version </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/review-comments/" class="md-nav__link"> <span class="md-ellipsis"> Review Comments </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/activity-trail/" class="md-nav__link"> <span class="md-ellipsis"> Who did what, when — reading a project's activity </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/admin-scopes/" class="md-nav__link"> <span class="md-ellipsis"> Renaming a project, and what can never change </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/authority-view/" class="md-nav__link"> <span class="md-ellipsis"> Who holds each function here, and what may it do </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/break-glass/" class="md-nav__link"> <span class="md-ellipsis"> The permission I need is held by a role I do not hold — what now? </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/coverage-and-chains/" class="md-nav__link"> <span class="md-ellipsis"> Coverage or chains? — which requirement view answers your question </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/credentials/" class="md-nav__link"> <span class="md-ellipsis"> Your password: changing it, resetting it — and why LQMS never sees it </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/emergency-cover/" class="md-nav__link"> <span class="md-ellipsis"> Who may cover for whom here — declaring emergency cover, and reading where the glass was broken </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/key-rules/" class="md-nav__link"> <span class="md-ellipsis"> Special elements: what a project declares, and when its names freeze </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/memberships/" class="md-nav__link"> <span class="md-ellipsis"> Belonging to an organization: invitations, acceptance, and what they change </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/provisioning-birth-path/" class="md-nav__link"> <span class="md-ellipsis"> How a new organization and project are born </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/review-policies/" class="md-nav__link"> <span class="md-ellipsis"> What a review policy controls, and what an empty grant means </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/setup-wizard-lanes/" class="md-nav__link"> <span class="md-ellipsis"> The setup wizard: the lanes, and what it writes </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/staffing-and-assignments/" class="md-nav__link"> <span class="md-ellipsis"> Staffing people into roles, and the accountable holder </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/topics/technical-dossier/" class="md-nav__link"> <span class="md-ellipsis"> How do I assemble a technical dossier? </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/trace-items/" class="md-nav__link"> <span class="md-ellipsis"> Trace Items — Requirements, Risks & Risk Controls </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/trace-links/" class="md-nav__link"> <span class="md-ellipsis"> Trace Links & the Suspect Cycle </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/traceability-workspace/" class="md-nav__link"> <span class="md-ellipsis"> The Traceability Workspace </span> </a> </li> <li class="md-nav__item"> <a href="../../user-guide/training/" class="md-nav__link"> <span class="md-ellipsis"> Training & Acknowledgements </span> </a> </li> </ul> </nav> </li> </ul> </nav> </div> </div> </div> <div class="md-sidebar md-sidebar--secondary" data-md-component="sidebar" data-md-type="toc" > <div class="md-sidebar__scrollwrap"> <div class="md-sidebar__inner"> <nav class="md-nav md-nav--secondary" aria-label="Table of contents"> <label class="md-nav__title" for="__toc"> <span class="md-nav__icon md-icon"></span> Table of contents </label> <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix> <li class="md-nav__item"> <a href="#1-topology" class="md-nav__link"> <span class="md-ellipsis"> 1. Topology </span> </a> <nav class="md-nav" aria-label="1. Topology"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#why-keycloak-sits-on-a-subdomain-not-a-path" class="md-nav__link"> <span class="md-ellipsis"> Why Keycloak sits on a subdomain, not a path </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#2-sizing" class="md-nav__link"> <span class="md-ellipsis"> 2. Sizing </span> </a> </li> <li class="md-nav__item"> <a href="#3-prerequisites" class="md-nav__link"> <span class="md-ellipsis"> 3. Prerequisites </span> </a> <nav class="md-nav" aria-label="3. Prerequisites"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#31-dns--two-records-before-the-first-start" class="md-nav__link"> <span class="md-ellipsis"> 3.1 DNS — two records, before the first start </span> </a> </li> <li class="md-nav__item"> <a href="#32-firewall" class="md-nav__link"> <span class="md-ellipsis"> 3.2 Firewall </span> </a> </li> <li class="md-nav__item"> <a href="#33-unattended-os-updates" class="md-nav__link"> <span class="md-ellipsis"> 3.3 Unattended OS updates </span> </a> </li> <li class="md-nav__item"> <a href="#34-the-deploy-files-on-the-server--delivered-by-scp-not-by-git" class="md-nav__link"> <span class="md-ellipsis"> 3.4 The deploy files on the server — delivered by scp, not by git </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#4-the-image--published-by-ci-pulled-by-the-server" class="md-nav__link"> <span class="md-ellipsis"> 4. The image — published by CI, pulled by the server </span> </a> <nav class="md-nav" aria-label="4. The image — published by CI, pulled by the server"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#41-the-package-is-private--confirm-it-once" class="md-nav__link"> <span class="md-ellipsis"> 4.1 The package is private — confirm it once </span> </a> </li> <li class="md-nav__item"> <a href="#42-the-servers-registry-credential--a-classic-pat-with-readpackages" class="md-nav__link"> <span class="md-ellipsis"> 4.2 The server's registry credential — a classic PAT with read:packages </span> </a> </li> <li class="md-nav__item"> <a href="#43-before-you-cut-the-release-tag-the-deploy-runtime-render-check" class="md-nav__link"> <span class="md-ellipsis"> 4.3 Before you cut the release tag: the deploy-runtime render check </span> </a> </li> <li class="md-nav__item"> <a href="#44-if-you-must-build-it-yourself" class="md-nav__link"> <span class="md-ellipsis"> 4.4 If you must build it yourself </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#5-secrets" class="md-nav__link"> <span class="md-ellipsis"> 5. Secrets </span> </a> <nav class="md-nav" aria-label="5. Secrets"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#51-inventory--what-must-be-regenerated" class="md-nav__link"> <span class="md-ellipsis"> 5.1 Inventory — what must be regenerated </span> </a> </li> <li class="md-nav__item"> <a href="#52-how-the-realm-gets-them-vardefault-substitution" class="md-nav__link"> <span class="md-ellipsis"> 5.2 How the realm gets them: ${VAR:default} substitution </span> </a> </li> <li class="md-nav__item"> <a href="#53-inviting-people-who-have-no-account-yet--mail-and-the-one-credential-that-mints-logins" class="md-nav__link"> <span class="md-ellipsis"> 5.3 Inviting people who have no account yet — mail, and the one credential that mints logins </span> </a> <nav class="md-nav" aria-label="5.3 Inviting people who have no account yet — mail, and the one credential that mints logins"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#existing-installations-the-realm-was-imported-once-so-add-the-client-by-hand" class="md-nav__link"> <span class="md-ellipsis"> Existing installations: the realm was imported once, so add the client by hand </span> </a> </li> <li class="md-nav__item"> <a href="#existing-installations-credential-self-service-register-2026-08-06-27" class="md-nav__link"> <span class="md-ellipsis"> Existing installations: credential self-service (register 2026-08-06 #27) </span> </a> </li> </ul> </nav> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#6-bring-up" class="md-nav__link"> <span class="md-ellipsis"> 6. Bring-up </span> </a> <nav class="md-nav" aria-label="6. Bring-up"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#61-first-content-wizard-birth-or-demo-seed" class="md-nav__link"> <span class="md-ellipsis"> 6.1 First content: wizard birth or demo seed </span> </a> </li> <li class="md-nav__item"> <a href="#62-existing-installations-give-the-tenant-its-real-name" class="md-nav__link"> <span class="md-ellipsis"> 6.2 Existing installations: give the tenant its real name </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#7-do-not-run-the-demo-scripts-against-a-server-stack" class="md-nav__link"> <span class="md-ellipsis"> 7. Do not run the demo scripts against a server stack </span> </a> <nav class="md-nav" aria-label="7. Do not run the demo scripts against a server stack"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#71-base-file-only-scripts-and-the-pulled-image" class="md-nav__link"> <span class="md-ellipsis"> 7.1 Base-file-only scripts and the pulled image </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#8-keycloak-admin-access" class="md-nav__link"> <span class="md-ellipsis"> 8. Keycloak admin access </span> </a> </li> <li class="md-nav__item"> <a href="#8b-importing-a-corpus--the-one-shot-rule-read-before-the-production-import" class="md-nav__link"> <span class="md-ellipsis"> 8b. Importing a corpus — the one-shot rule (read before the production import) </span> </a> </li> <li class="md-nav__item"> <a href="#8c-exporting-a-corpus--the-portability-bundle-adr-0125" class="md-nav__link"> <span class="md-ellipsis"> 8c. Exporting a corpus — the portability bundle (ADR-0125) </span> </a> </li> <li class="md-nav__item"> <a href="#9-backups" class="md-nav__link"> <span class="md-ellipsis"> 9. Backups </span> </a> <nav class="md-nav" aria-label="9. Backups"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#91-the-schedule--what-the-reference-installation-actually-runs" class="md-nav__link"> <span class="md-ellipsis"> 9.1 The schedule — what the reference installation actually runs </span> </a> </li> <li class="md-nav__item"> <a href="#92-the-off-server-copy-is-the-point" class="md-nav__link"> <span class="md-ellipsis"> 9.2 The off-server copy is the point </span> </a> </li> <li class="md-nav__item"> <a href="#93-restore-rehearsal--preferably-not-on-this-box" class="md-nav__link"> <span class="md-ellipsis"> 9.3 Restore rehearsal — preferably not on this box </span> </a> <nav class="md-nav" aria-label="9.3 Restore rehearsal — preferably not on this box"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#what-a-rehearsal-record-proves--and-why-the-pre-2026-08-19-rows-do-not" class="md-nav__link"> <span class="md-ellipsis"> What a rehearsal record proves — and why the pre-2026-08-19 rows do not </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#94-restoring-onto-the-server--the-exact-sequence" class="md-nav__link"> <span class="md-ellipsis"> 9.4 Restoring onto the server — the exact sequence </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#10-rotating-a-secret-after-first-boot" class="md-nav__link"> <span class="md-ellipsis"> 10. Rotating a secret after first boot </span> </a> </li> <li class="md-nav__item"> <a href="#11-keeping-it-current" class="md-nav__link"> <span class="md-ellipsis"> 11. Keeping it current </span> </a> <nav class="md-nav" aria-label="11. Keeping it current"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#111-upgrading-lqms--the-whole-ritual" class="md-nav__link"> <span class="md-ellipsis"> 11.1 Upgrading LQMS — the whole ritual </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#12-first-deploy-checklist" class="md-nav__link"> <span class="md-ellipsis"> 12. First-deploy checklist </span> </a> </li> <li class="md-nav__item"> <a href="#13-troubleshooting" class="md-nav__link"> <span class="md-ellipsis"> 13. Troubleshooting </span> </a> <nav class="md-nav" aria-label="13. Troubleshooting"> <ul class="md-nav__list"> <li class="md-nav__item"> <a href="#131-issuer-mismatch--the-classic-breakage" class="md-nav__link"> <span class="md-ellipsis"> 13.1 Issuer mismatch — the classic breakage </span> </a> </li> <li class="md-nav__item"> <a href="#132-login-loop--the-session-cookie-is-dropped" class="md-nav__link"> <span class="md-ellipsis"> 13.2 Login loop — the session cookie is dropped </span> </a> </li> <li class="md-nav__item"> <a href="#133-certificates-are-not-issued" class="md-nav__link"> <span class="md-ellipsis"> 13.3 Certificates are not issued </span> </a> </li> <li class="md-nav__item"> <a href="#134-exec-format-error-when-the-app-container-starts" class="md-nav__link"> <span class="md-ellipsis"> 13.4 exec format error when the app container starts </span> </a> </li> <li class="md-nav__item"> <a href="#135-the-app-refuses-to-boot-fan-out-deployment-check" class="md-nav__link"> <span class="md-ellipsis"> 13.5 The app refuses to boot: fan-out deployment check </span> </a> </li> <li class="md-nav__item"> <a href="#136-database-authentication-failure-after-a-restore" class="md-nav__link"> <span class="md-ellipsis"> 13.6 Database authentication failure after a restore </span> </a> </li> <li class="md-nav__item"> <a href="#137-http-431--a-wedged-browser-profile" class="md-nav__link"> <span class="md-ellipsis"> 13.7 HTTP 431 / a wedged browser profile </span> </a> </li> <li class="md-nav__item"> <a href="#138-the-image-will-not-pull" class="md-nav__link"> <span class="md-ellipsis"> 13.8 The image will not pull </span> </a> </li> </ul> </nav> </li> <li class="md-nav__item"> <a href="#14-what-this-deployment-deliberately-does-not-do" class="md-nav__link"> <span class="md-ellipsis"> 14. What this deployment deliberately does not do </span> </a> </li> </ul> </nav> </div> </div> </div> <div class="md-content" data-md-component="content"> <article class="md-content__inner md-typeset"> <a href="https://github.com/nburri/lqms/edit/main/docs/operations/server-deployment.md" title="Edit this page" class="md-content__button md-icon"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M10 20H6V4h7v5h5v3.1l2-2V8l-6-6H6c-1.1 0-2 .9-2 2v16c0 1.1.9 2 2 2h4zm10.2-7c.1 0 .3.1.4.2l1.3 1.3c.2.2.2.6 0 .8l-1 1-2.1-2.1 1-1c.1-.1.2-.2.4-.2m0 3.9L14.1 23H12v-2.1l6.1-6.1z"/></svg> </a> <h1 id="lqms-server-deployment-single-linux-host-public-https">LQMS Server Deployment (single Linux host, public HTTPS)<a class="headerlink" href="#lqms-server-deployment-single-linux-host-public-https" title="Permanent link">¶</a></h1> <p>How to run LQMS on a real server: one small Linux box, Docker Compose, a public domain, automatic TLS. This is the <strong>topology</strong> guide — it says what to expose, what to keep private, which secrets to generate and in what order to bring things up.</p> <p>Two companion documents, both still authoritative for what they cover:</p> <table> <thead> <tr> <th>Document</th> <th>Covers</th> </tr> </thead> <tbody> <tr> <td><a href="../deployment/"><code>docs/operations/deployment.md</code></a></td> <td>the <strong>configuration reference</strong> — every <code>quarkus.*</code> / <code>lqms.*</code> key, the BYPASSRLS requirement, IdP integration, upgrade path, bootstrap flow. Read §4 (database roles) and §5 (configuration) alongside this file.</td> </tr> <tr> <td><a href="https://github.com/nburri/lqms/blob/main/deploy/README.md"><code>deploy/README.md</code></a></td> <td>the <strong>localhost demo stack</strong> — build prerequisites, the login theme, SBOM, backup/restore scripts.</td> </tr> </tbody> </table> <p>The files this guide drives:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-0-1" name="__codelineno-0-1" href="#__codelineno-0-1"></a>deploy/docker-compose.yml # the base stack (unchanged; never used alone on a server) <a id="__codelineno-0-2" name="__codelineno-0-2" href="#__codelineno-0-2"></a>deploy/docker-compose.server.yml # the server OVERRIDE — pull-based app, Caddy, TLS, no public ports <a id="__codelineno-0-3" name="__codelineno-0-3" href="#__codelineno-0-3"></a>deploy/Caddyfile # the reverse proxy: routes, HSTS, admin-console block <a id="__codelineno-0-4" name="__codelineno-0-4" href="#__codelineno-0-4"></a>deploy/server.env.example # every secret to generate + LQMS_VERSION (the release to run) <a id="__codelineno-0-5" name="__codelineno-0-5" href="#__codelineno-0-5"></a>deploy/lqms-realm.json # the realm; every credential in it is env-substitutable <a id="__codelineno-0-6" name="__codelineno-0-6" href="#__codelineno-0-6"></a>.github/workflows/ci.yml # the `publish-image` job — the ONLY producer of the image </code></pre></div> <p><strong>The shape of a deployment, in one sentence:</strong> the server runs a <strong>published image pulled from <code>ghcr.io/nburri/lqms</code></strong> and a <strong>copy of <code>deploy/</code> delivered by <code>scp</code>/<code>rsync</code></strong>. It does not clone the repository, does not hold a toolchain, and never builds. §3.4 covers the files, §4 the image.</p> <hr /> <h2 id="1-topology">1. Topology<a class="headerlink" href="#1-topology" title="Permanent link">¶</a></h2> <pre class="mermaid"><code>flowchart LR U([Browser]) -->|"443 · https://qms.example.com"| C U -->|"443 · https://auth.qms.example.com"| C C[caddy<br/>auto-TLS, only public ports 80/443] C -->|"http, compose network"| A["app :8080<br/>(no published port)"] C -->|"http, compose network<br/>/admin* and /realms/master* → 404"| K["keycloak :8080<br/>(loopback publish 8081 only)"] A -->|"backchannel<br/>http://keycloak:8080"| K A --> P[("postgres :5432<br/>never published")] A --> V[["content volume<br/>(blobs, ADR-0001)"]] S([Operator]) -.->|"ssh -L 8081:localhost:8081"| K</code></pre> <p><strong>Exposure summary — this is the security claim of the deployment:</strong></p> <table> <thead> <tr> <th>Reachable from the internet</th> <th>Reachable from the host only</th> <th>Not published at all</th> </tr> </thead> <tbody> <tr> <td><code>443/tcp</code>, <code>443/udp</code> (HTTP/3), <code>80/tcp</code> (ACME + redirect to HTTPS)</td> <td>Keycloak on <code>127.0.0.1:8081</code> — the SSH-tunnel target for the admin console</td> <td>PostgreSQL, the app's HTTP port, Keycloak's management port (9000)</td> </tr> </tbody> </table> <p>The Keycloak <strong>admin console and master realm are not routed publicly</strong>: Caddy answers <code>404</code> for <code>/admin*</code> and <code>/realms/master*</code> on <code>auth.<domain></code> (a 404, not a 403 — no confirmation that an admin console lives there). See §8 for how you reach it instead.</p> <h3 id="why-keycloak-sits-on-a-subdomain-not-a-path">Why Keycloak sits on a subdomain, not a path<a class="headerlink" href="#why-keycloak-sits-on-a-subdomain-not-a-path" title="Permanent link">¶</a></h3> <p>Decided and stated here so it is not re-litigated at 2 a.m.:</p> <ul> <li>Keycloak 26's <code>--hostname</code> accepts a full URL, so a host root needs no path handling at all. Serving Keycloak under <code>/auth</code> instead requires <code>KC_HTTP_RELATIVE_PATH=/auth</code>, and the image's own help says of <code>--http-management-relative-path</code>: <em>"If not given, the value is inherited from HTTP options."</em> The management port's health endpoint would silently move to <code>/auth/health/ready</code> and the compose healthcheck in <code>docker-compose.yml</code> (which probes <code>/health/ready</code> on <code>:9000</code>) would break. Verified against the <code>quay.io/keycloak/keycloak:26.4</code> image.</li> <li>A separate origin keeps Keycloak's cookies (<code>AUTH_SESSION_ID</code>, <code>KEYCLOAK_IDENTITY</code>) off the app origin. They are host-only cookies, so they never join the app's <code>Cookie</code> header — which matters here because an oversized <code>Cookie</code> header is exactly the HTTP 431 failure mode the app's OIDC settings were hardened against (<code>application.properties</code>, "finding #18").</li> <li>The cost is <strong>one extra DNS record and one extra certificate</strong>, both handled automatically by Caddy. The <code>SameSite=lax</code> session posture is unaffected either way: the login redirect is a top-level GET navigation, which Lax cookies accompany.</li> </ul> <hr /> <h2 id="2-sizing">2. Sizing<a class="headerlink" href="#2-sizing" title="Permanent link">¶</a></h2> <p>Everything runs on one host: two JVMs (app, Keycloak), PostgreSQL, and a small Go proxy.</p> <table> <thead> <tr> <th>Resource</th> <th>Minimum</th> <th>Recommended</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>vCPU</td> <td>2</td> <td>4</td> <td>2 is enough at rest and for a handful of concurrent users; PDF/audit-pack rendering and the SPA build are the CPU spikes.</td> </tr> <tr> <td>RAM</td> <td>4 GB</td> <td><strong>8 GB</strong></td> <td>Two JVMs plus PostgreSQL. 4 GB works but leaves no headroom for a render burst or a restore rehearsal. The box does <strong>not</strong> build the image (§4), so the ~4 GB a Gradle + Angular build needs is not part of this budget.</td> </tr> <tr> <td>Disk</td> <td>40 GB SSD</td> <td>80 GB SSD</td> <td>OS + Docker images ≈ 6-8 GB; the database and the content blob store (ADR-0001) grow with the QMS; local backup archives default to a <strong>35-day</strong> retention window (<code>LQMS_BACKUP_RETENTION_DAYS</code>) and are the largest variable. Keep ≥ 30 % free — PostgreSQL and the backup staging directory both need room.</td> </tr> <tr> <td>Network</td> <td>any</td> <td>—</td> <td>Ports 80/443 inbound; outbound HTTPS for ACME and image pulls.</td> </tr> <tr> <td>OS</td> <td>Debian 12+ / Ubuntu 22.04+</td> <td>—</td> <td>Anything with a current Docker Engine and <code>unattended-upgrades</code>.</td> </tr> <tr> <td>Docker</td> <td>Engine 24+, Compose <strong>v2.24+</strong></td> <td>—</td> <td>v2.24 is the floor for the <code>!reset</code> / <code>!override</code> merge tags used by the server override. Check with <code>docker compose version</code>.</td> </tr> </tbody> </table> <p>Swap: give a 4 GB box at least 2 GB of swap so a JVM spike degrades instead of being OOM-killed.</p> <hr /> <h2 id="3-prerequisites">3. Prerequisites<a class="headerlink" href="#3-prerequisites" title="Permanent link">¶</a></h2> <h3 id="31-dns--two-records-before-the-first-start">3.1 DNS — two records, before the first start<a class="headerlink" href="#31-dns--two-records-before-the-first-start" title="Permanent link">¶</a></h3> <p>Caddy obtains certificates over the ACME <strong>HTTP-01</strong> challenge, which requires both names to already resolve to this server on port 80.</p> <table> <thead> <tr> <th>Record</th> <th>Type</th> <th>Value</th> </tr> </thead> <tbody> <tr> <td><code>qms.example.com</code></td> <td>A (and AAAA if the host has IPv6)</td> <td>the server's public IP</td> </tr> <tr> <td><code>auth.qms.example.com</code></td> <td>A (and AAAA)</td> <td>the server's public IP</td> </tr> </tbody> </table> <p><code>auth.<domain></code> is <strong>derived</strong> from the single <code>LQMS_DOMAIN</code> variable — you never configure it separately, but the record must exist. Verify before bringing the stack up:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-1-1" name="__codelineno-1-1" href="#__codelineno-1-1"></a>dig<span class="w"> </span>+short<span class="w"> </span>qms.example.com<span class="w"> </span>auth.qms.example.com </code></pre></div> <p>If the AAAA record exists but the host has no working IPv6 route, ACME validation can fail over IPv6 while succeeding over IPv4. Publish AAAA only when IPv6 actually works.</p> <h3 id="32-firewall">3.2 Firewall<a class="headerlink" href="#32-firewall" title="Permanent link">¶</a></h3> <p>Allow <code>22</code> (SSH), <code>80</code>, <code>443</code>. Deny the rest inbound.</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-2-1" name="__codelineno-2-1" href="#__codelineno-2-1"></a>ufw<span class="w"> </span>default<span class="w"> </span>deny<span class="w"> </span>incoming <a id="__codelineno-2-2" name="__codelineno-2-2" href="#__codelineno-2-2"></a>ufw<span class="w"> </span>default<span class="w"> </span>allow<span class="w"> </span>outgoing <a id="__codelineno-2-3" name="__codelineno-2-3" href="#__codelineno-2-3"></a>ufw<span class="w"> </span>allow<span class="w"> </span><span class="m">22</span>/tcp <a id="__codelineno-2-4" name="__codelineno-2-4" href="#__codelineno-2-4"></a>ufw<span class="w"> </span>allow<span class="w"> </span><span class="m">80</span>,443/tcp <a id="__codelineno-2-5" name="__codelineno-2-5" href="#__codelineno-2-5"></a>ufw<span class="w"> </span>allow<span class="w"> </span><span class="m">443</span>/udp<span class="w"> </span><span class="c1"># HTTP/3; harmless to omit</span> <a id="__codelineno-2-6" name="__codelineno-2-6" href="#__codelineno-2-6"></a>ufw<span class="w"> </span><span class="nb">enable</span> </code></pre></div> <blockquote> <p><strong>Docker publishes ports around ufw.</strong> Docker inserts its own NAT/forward rules, so a container port published on <code>0.0.0.0</code> is reachable <strong>even if ufw denies it</strong>. This deployment publishes only 80/443 plus a loopback-bound 8081, so nothing leaks — but the moment you run something that publishes another port (notably <code>verify-backup.sh</code>, §9.3), that port is public. Defence in depth, scoped to the external interface so container egress is unaffected:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-3-1" name="__codelineno-3-1" href="#__codelineno-3-1"></a><span class="c1"># substitute the real external interface (ip -o -4 route show to default | awk '{print $5}')</span> <a id="__codelineno-3-2" name="__codelineno-3-2" href="#__codelineno-3-2"></a>iptables<span class="w"> </span>-I<span class="w"> </span>DOCKER-USER<span class="w"> </span>-i<span class="w"> </span>eth0<span class="w"> </span>-p<span class="w"> </span>tcp<span class="w"> </span>-m<span class="w"> </span>conntrack<span class="w"> </span>--ctstate<span class="w"> </span>NEW<span class="w"> </span><span class="se">\</span> <a id="__codelineno-3-3" name="__codelineno-3-3" href="#__codelineno-3-3"></a><span class="w"> </span>-m<span class="w"> </span>multiport<span class="w"> </span>!<span class="w"> </span>--dports<span class="w"> </span><span class="m">80</span>,443<span class="w"> </span>-j<span class="w"> </span>DROP <a id="__codelineno-3-4" name="__codelineno-3-4" href="#__codelineno-3-4"></a>apt<span class="w"> </span>install<span class="w"> </span>iptables-persistent<span class="w"> </span><span class="c1"># or the rule is gone at the next reboot</span> </code></pre></div> </blockquote> <h3 id="33-unattended-os-updates">3.3 Unattended OS updates<a class="headerlink" href="#33-unattended-os-updates" title="Permanent link">¶</a></h3> <p>Security patches for the host — <strong>not</strong> for the container images (see §11 for those).</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-4-1" name="__codelineno-4-1" href="#__codelineno-4-1"></a>apt<span class="w"> </span>install<span class="w"> </span>unattended-upgrades<span class="w"> </span>apt-listchanges <a id="__codelineno-4-2" name="__codelineno-4-2" href="#__codelineno-4-2"></a>dpkg-reconfigure<span class="w"> </span>-plow<span class="w"> </span>unattended-upgrades<span class="w"> </span><span class="c1"># answer "yes"</span> <a id="__codelineno-4-3" name="__codelineno-4-3" href="#__codelineno-4-3"></a><span class="c1"># optional but recommended: automatic reboot in a quiet window</span> <a id="__codelineno-4-4" name="__codelineno-4-4" href="#__codelineno-4-4"></a><span class="c1"># /etc/apt/apt.conf.d/50unattended-upgrades</span> <a id="__codelineno-4-5" name="__codelineno-4-5" href="#__codelineno-4-5"></a><span class="c1"># Unattended-Upgrade::Automatic-Reboot "true";</span> <a id="__codelineno-4-6" name="__codelineno-4-6" href="#__codelineno-4-6"></a><span class="c1"># Unattended-Upgrade::Automatic-Reboot-Time "04:30";</span> </code></pre></div> <p>A reboot restarts the stack automatically: every service in the server override carries <code>restart: unless-stopped</code>.</p> <h3 id="34-the-deploy-files-on-the-server--delivered-by-scp-not-by-git">3.4 The deploy files on the server — delivered by <code>scp</code>, not by <code>git</code><a class="headerlink" href="#34-the-deploy-files-on-the-server--delivered-by-scp-not-by-git" title="Permanent link">¶</a></h3> <p><strong>The server does not get a clone.</strong> It gets a copy of <code>deploy/</code>, and nothing else: no sources, no Gradle wrapper, no <code>.git</code>. That is the whole point of the pull-based image (§4) — the box holds configuration and data, never a build.</p> <p>The reference installation (lqms.ch) runs the stack as an unprivileged user, <strong><code>lqms-ops</code></strong>, and keeps the files in that user's home. <strong><code>/home/lqms-ops/lqms/deploy</code></strong> is therefore the path used throughout this guide, and the same user owns the registry credential (§4.2) and the backup crontab (§9.1). From your checkout:</p> <blockquote> <p><strong>--delete hazard (found live, 2026-08-23):</strong> without the <code>*.bak-*</code> / <code>*.log*</code> exclusions below, this sync would delete the server-side <code>server.env.bak-*</code> safety copies, script <code>.bak-*</code> preservation files, and <code>backup-verification.log*</code> — operational evidence, not repo files. Also learned the hard way the same day: deploy-script FIXES only reach the box when this sync actually runs — restore.sh and verify-backup.sh sat drifted (pre-fix, vacuous-pass era) on the server for a week. Run the sync as part of every release's server session. (And a second lesson from the v0.9.19 run: the first version of THIS note was spliced inside the code fence by a careless edit, so the printed command did not paste-run. The fence below is the reconstructed command the run actually used, verified live with zero deletions.)</p> </blockquote> <div class="highlight"><pre><span></span><code><a id="__codelineno-5-1" name="__codelineno-5-1" href="#__codelineno-5-1"></a>rsync<span class="w"> </span>-av<span class="w"> </span>--delete<span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-2" name="__codelineno-5-2" href="#__codelineno-5-2"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'server.env.bak-*'</span><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'*.bak-*'</span><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'*.log'</span><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'*.log.*'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-3" name="__codelineno-5-3" href="#__codelineno-5-3"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'server.env'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-4" name="__codelineno-5-4" href="#__codelineno-5-4"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'lqms-compose'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-5" name="__codelineno-5-5" href="#__codelineno-5-5"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'seed-local.d/'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-6" name="__codelineno-5-6" href="#__codelineno-5-6"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'backups/'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-7" name="__codelineno-5-7" href="#__codelineno-5-7"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'backup-verification.log'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-8" name="__codelineno-5-8" href="#__codelineno-5-8"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'smoke.sh'</span><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'nightly-smoke.sh'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-9" name="__codelineno-5-9" href="#__codelineno-5-9"></a><span class="w"> </span>--exclude<span class="w"> </span><span class="s1">'.DS_Store'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-5-10" name="__codelineno-5-10" href="#__codelineno-5-10"></a><span class="w"> </span>deploy/<span class="w"> </span>lqms-ops@server:/home/lqms-ops/lqms/deploy/ </code></pre></div> <p>Read the exclusions — each one is deliberate:</p> <table> <thead> <tr> <th>Excluded</th> <th>Why</th> </tr> </thead> <tbody> <tr> <td><code>server.env</code></td> <td>it holds every secret of the deployment. It is created <strong>on the server</strong> (§5) and never travels from a laptop as a side effect of an update.</td> </tr> <tr> <td><code>lqms-compose</code></td> <td>the wrapper (§6) is created <strong>on the server</strong> and is not in the repo — without this exclusion, <code>--delete</code> removes it (it did, 2026-08-16; the v0.9.12 deploy recreated it by hand).</td> </tr> <tr> <td><code>seed-local.d/</code></td> <td>local scratch. Nothing about it belongs on a server.</td> </tr> <tr> <td><code>backups/</code>, <code>backup-verification.log</code></td> <td>these are the <strong>server's own</strong> output. Pushing your laptop's copies over them would destroy the archive set.</td> </tr> <tr> <td><code>smoke.sh</code>, <code>nightly-smoke.sh</code></td> <td>the two scripts that are dangerous <strong>by construction</strong> on a server. <code>smoke.sh</code> ends in <code>docker compose -p lqms down -v</code>, which deletes the database, the content blobs and the Keycloak volume; <code>nightly-smoke.sh</code> builds at HEAD, which a box with no repository cannot do anyway. The safest place for both is <em>not on the box</em> (§7).</td> </tr> </tbody> </table> <blockquote> <p><strong><code>seed-demo.sh</code> and <code>DEMO.md</code> are deliberately NOT excluded — they are on the box, and that is what the reference installation looks like.</strong> Presence is not permission. Unlike the two above, <code>seed-demo.sh</code> cannot quietly succeed here: it authenticates with the demo client secret, the demo Keycloak master password and demo user passwords, all of which a server has replaced, so it fails early rather than half-way (§7). Excluding it would buy a false sense of safety and hide the rule that actually protects the installation — <em>never run it against a server stack</em> — which holds whether the file is present or not. <code>DEMO.md</code> is the localhost walk-through that describes it; on a server it is inert text.</p> <p><code>--exclude</code> also protects a path from <code>--delete</code>, so the server's <code>server.env</code> and <code>backups/</code> survive every subsequent sync. Run the same command to update the deploy files; it is idempotent.</p> </blockquote> <p>What legitimately lands on the server: both compose files, <code>Caddyfile</code>, <code>lqms-realm.json</code>, <code>postgres-init.sql</code>, <code>keycloak-theme/</code>, <code>lqms-init.sh</code> (the first-deploy interview, §5), <code>backup.sh</code>, <code>restore.sh</code>, <code>verify-backup.sh</code>, <code>server.env.example</code> — plus <code>seed-demo.sh</code> and <code>DEMO.md</code> as described above. <code>Dockerfile</code> rides along harmlessly — its build context (the repository root) does not exist there, which is exactly the intended dead end; <code>README.md</code> and the remaining small helpers are inert.</p> <p>Ownership: the deploy directory belongs to the user that runs the stack — <code>chown -R lqms-ops:lqms-ops /home/lqms-ops/lqms/deploy</code> on the reference installation. Keep that the <em>same</em> user that holds the <code>docker login</code> (§4.2) and the backup crontab (§9.1); splitting them is how you get a pull that works by hand and fails from cron. <code>server.env</code> stays <code>chmod 600</code>.</p> <hr /> <h2 id="4-the-image--published-by-ci-pulled-by-the-server">4. The image — published by CI, pulled by the server<a class="headerlink" href="#4-the-image--published-by-ci-pulled-by-the-server" title="Permanent link">¶</a></h2> <p><strong>The server pulls. It never builds.</strong> The image is produced exactly once, by the <code>publish-image</code> job in <a href="https://github.com/nburri/lqms/blob/main/.github/workflows/ci.yml"><code>.github/workflows/ci.yml</code></a>, and pushed to the GitHub Container Registry:</p> <table> <thead> <tr> <th></th> <th></th> </tr> </thead> <tbody> <tr> <td>Registry path</td> <td><code>ghcr.io/nburri/lqms</code></td> </tr> <tr> <td>Published on</td> <td>a <strong>release tag push</strong> (<code>v*</code>), and on manual <code>workflow_dispatch</code></td> </tr> <tr> <td>Tags written</td> <td><code>:vX.Y.Z</code> for the release, plus <code>:latest</code> moved onto it. A dispatch that does <strong>not</strong> name a tag publishes <code>:<ref>-<sha></code> and does <strong>not</strong> move <code>:latest</code></td> </tr> <tr> <td>Platform</td> <td><strong><code>linux/amd64</code> only</strong> — built natively on the runner</td> </tr> <tr> <td>Visibility</td> <td><strong>private</strong> (see below)</td> </tr> <tr> <td>Build stamps</td> <td><code>GIT_SHA</code> and <code>PRODUCT_VERSION</code> (<code>git describe</code>, ADR-0090) baked in; they surface at <code>/api/version</code></td> </tr> </tbody> </table> <p>This retires the architecture trap that used to lead this section. CI builds on an amd64 runner, so <code>exec format error</code> on an x86_64 VPS cannot happen from a laptop cross-build any more (§13.4 keeps the symptom, for the case where somebody builds by hand anyway). arm64 is deliberately not published — the workflow comments carry the reason and the one-line change if a target ever needs it.</p> <h3 id="41-the-package-is-private--confirm-it-once">4.1 The package is private — confirm it once<a class="headerlink" href="#41-the-package-is-private--confirm-it-once" title="Permanent link">¶</a></h3> <p>A container package published to ghcr.io is <strong>private by default</strong>, and its visibility is a property of the <em>package</em>, not of the repository: the package inherits the linked repository's <strong>access permissions</strong>, but not its visibility, and a later change to the repository does not reliably flip it. <code>nburri/lqms</code> is a private repository today, which is consistent — but "consistent by accident" is not a control.</p> <p>So do this once, after the first publish, and record it:</p> <blockquote> <p><strong><a href="https://github.com/nburri/lqms/pkgs/container/lqms">https://github.com/nburri/lqms/pkgs/container/lqms</a></strong> → the landing page must say <strong>Private</strong>. To change it: on the package landing page, right-hand side → <strong>Package settings</strong> → bottom of the page, <strong>Danger Zone</strong> → <strong>Change visibility</strong>. (Also reachable from your profile → <strong>Packages</strong> → <code>lqms</code> → <strong>Package settings</strong>.)</p> </blockquote> <p>If the repository is ever made public, re-check this page. A public LQMS image is a supply-chain and licensing decision, not a side effect.</p> <h3 id="42-the-servers-registry-credential--a-classic-pat-with-readpackages">4.2 The server's registry credential — a classic PAT with <code>read:packages</code><a class="headerlink" href="#42-the-servers-registry-credential--a-classic-pat-with-readpackages" title="Permanent link">¶</a></h3> <p>Pulling a private package needs an authenticated Docker client. <strong>GitHub Packages supports only a personal access token (classic)</strong> for the container registry — a <em>fine-grained</em> token does not authenticate to <code>ghcr.io</code>. Verified against GitHub's own "Working with the Container registry" documentation; do not spend an evening on a fine-grained token that will keep returning <code>denied</code>.</p> <p><strong>Create the token</strong> (once, on your own account):</p> <ol> <li>GitHub → avatar → <strong>Settings</strong> → <strong>Developer settings</strong> → <strong>Personal access tokens</strong> → <strong>Tokens (classic)</strong> → <strong>Generate new token (classic)</strong>.</li> <li>Note: <code>lqms server pull</code> (name it after the box it lives on).</li> <li>Expiration: pick a real date and <strong>put it in your calendar</strong>. An expired token does not take the site down — the containers keep running — but the next <code>pull</code> fails with <code>denied</code>, usually in the middle of an upgrade.</li> <li>Scopes: tick <strong><code>read:packages</code></strong> and nothing else. Not <code>write:packages</code>, not <code>repo</code>.</li> </ol> <p><strong>Use it on the server:</strong></p> <div class="highlight"><pre><span></span><code><a id="__codelineno-6-1" name="__codelineno-6-1" href="#__codelineno-6-1"></a><span class="c1"># on the server, as the user that runs the stack — `lqms-ops` on the reference installation, the</span> <a id="__codelineno-6-2" name="__codelineno-6-2" href="#__codelineno-6-2"></a><span class="c1"># same user that owns /home/lqms-ops/lqms/deploy (§3.4) and the backup crontab (§9.1)</span> <a id="__codelineno-6-3" name="__codelineno-6-3" href="#__codelineno-6-3"></a><span class="nb">echo</span><span class="w"> </span><span class="s1">'<the-token>'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>docker<span class="w"> </span>login<span class="w"> </span>ghcr.io<span class="w"> </span>-u<span class="w"> </span>nburri<span class="w"> </span>--password-stdin <a id="__codelineno-6-4" name="__codelineno-6-4" href="#__codelineno-6-4"></a><span class="c1"># -> Login Succeeded</span> </code></pre></div> <p>Then prove the credential actually pulls, before you need it:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-7-1" name="__codelineno-7-1" href="#__codelineno-7-1"></a>docker<span class="w"> </span>pull<span class="w"> </span>ghcr.io/nburri/lqms:v0.9.0 <a id="__codelineno-7-2" name="__codelineno-7-2" href="#__codelineno-7-2"></a>docker<span class="w"> </span>image<span class="w"> </span>inspect<span class="w"> </span>ghcr.io/nburri/lqms:v0.9.0<span class="w"> </span>--format<span class="w"> </span><span class="s1">'{{.Architecture}}'</span><span class="w"> </span><span class="c1"># -> amd64</span> </code></pre></div> <p>Three things to know about that credential:</p> <ul> <li><strong>It is stored, not encrypted.</strong> <code>docker login</code> writes the token base64-encoded into <code>~/.docker/config.json</code> of the user that ran it. <code>chmod 600 ~/.docker/config.json</code>, and remember that the file is <em>per user</em> — a stack run as <code>lqms-ops</code> reads <code>/home/lqms-ops/.docker/config.json</code>, so a <code>docker login</code> performed as root or as your own account does nothing for it. This is the usual cause of a <code>pull</code> that works when you type it and fails from cron. <code>docker logout ghcr.io</code> removes it.</li> <li><strong>The scope is account-wide.</strong> A classic PAT's <code>read:packages</code> grants read to every package the account can read; GHCR offers no narrower token. On a single-owner account that is the practical minimum. When this stops being a single-owner setup, the standard hardening is a dedicated machine account with read access to this repository only, holding its own classic PAT.</li> <li><strong>The running stack does not need it.</strong> It is required to <code>pull</code> — first deploy, upgrades, and any restart that has to re-fetch a pruned image. A reboot of an unchanged stack does not pull (<code>restart: unless-stopped</code> restarts existing containers).</li> </ul> <h3 id="43-before-you-cut-the-release-tag-the-deploy-runtime-render-check">4.3 Before you cut the release tag: the deploy-runtime render check<a class="headerlink" href="#43-before-you-cut-the-release-tag-the-deploy-runtime-render-check" title="Permanent link">¶</a></h3> <p>The runtime stage is <code>eclipse-temurin:25-jre-noble</code> and runs as a non-root user. The deploy JVM (Temurin 25) differs from CI's JDK 21, and <strong>no CI job runs the deploy image</strong> — <code>publish-image</code> builds it, nothing exercises it. The PDF/audit-pack render path on that runtime is covered only by <code>deploy/nightly-smoke.sh</code> (the FWK005 guard). Run it <strong>on your build machine, before tagging a release</strong>, never on the server.</p> <h3 id="44-if-you-must-build-it-yourself">4.4 If you must build it yourself<a class="headerlink" href="#44-if-you-must-build-it-yourself" title="Permanent link">¶</a></h3> <p>Kept as a dead end on purpose: the supported path is pull. If a registry outage or an air-gapped target forces a local build, the mechanics are in <a href="https://github.com/nburri/lqms/blob/main/deploy/README.md"><code>deploy/README.md</code></a> (prerequisites: pre-generated jOOQ sources via <code>./gradlew :backend:persistence:generateJooq</code>, plus <code>GIT_SHA</code>/<code>PRODUCT_VERSION</code> as build args) — cross-build with <code>docker buildx build --platform linux/amd64 -f deploy/Dockerfile -t ghcr.io/nburri/lqms:<tag> .</code>, then <code>docker save … | ssh server 'docker load'</code> under the <em>same</em> tag <code>LQMS_VERSION</code> names. Under QEMU on an Apple Silicon Mac that build is slow and memory-hungry; on the server itself it needs the ~4 GB of free RAM the sizing table no longer budgets for.</p> <hr /> <h2 id="5-secrets">5. Secrets<a class="headerlink" href="#5-secrets" title="Permanent link">¶</a></h2> <p>Every credential in the demo stack is a well-known default. The server override refuses to start until each one is replaced: the variables are declared <code>${VAR:?…}</code>, so <code>docker compose</code> fails with the name of the missing variable rather than silently falling back.</p> <p>The first-deploy interview writes the whole file for you — it asks for the domain, the release tag, the organization's name and the admin identity, verifies both DNS records before anything can hit the Caddy HTTP-01 gate, and generates every secret in place (never typed, never echoed):</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-8-1" name="__codelineno-8-1" href="#__codelineno-8-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-8-2" name="__codelineno-8-2" href="#__codelineno-8-2"></a>./lqms-init.sh<span class="w"> </span><span class="c1"># refuses to overwrite an existing server.env</span> </code></pre></div> <p>The manual path remains, and fills the same rows:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-9-1" name="__codelineno-9-1" href="#__codelineno-9-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-9-2" name="__codelineno-9-2" href="#__codelineno-9-2"></a>cp<span class="w"> </span>server.env.example<span class="w"> </span>server.env <a id="__codelineno-9-3" name="__codelineno-9-3" href="#__codelineno-9-3"></a>chmod<span class="w"> </span><span class="m">600</span><span class="w"> </span>server.env <a id="__codelineno-9-4" name="__codelineno-9-4" href="#__codelineno-9-4"></a><span class="nv">$EDITOR</span><span class="w"> </span>server.env<span class="w"> </span><span class="c1"># generate each value with the command shown in the file</span> </code></pre></div> <h3 id="51-inventory--what-must-be-regenerated">5.1 Inventory — what must be regenerated<a class="headerlink" href="#51-inventory--what-must-be-regenerated" title="Permanent link">¶</a></h3> <table> <thead> <tr> <th>Variable</th> <th>Replaces (demo default)</th> <th>Where it lands</th> </tr> </thead> <tbody> <tr> <td><code>LQMS_POSTGRES_PASSWORD</code></td> <td><code>postgres</code> / <code>postgres</code></td> <td>PostgreSQL superuser</td> </tr> <tr> <td><code>LQMS_MIGRATION_PASSWORD</code></td> <td><code>lqms_migration</code> / <code>lqms_migration</code></td> <td>the <code>lqms_migration</code> DB role — created by <code>postgres-init.sql</code> at first boot <strong>and</strong> used as <code>QUARKUS_DATASOURCE_PASSWORD</code>. One variable, so the two cannot drift</td> </tr> <tr> <td><code>LQMS_KC_ADMIN_PASSWORD</code></td> <td><code>admin</code> / <code>admin</code></td> <td>Keycloak master-realm admin</td> </tr> <tr> <td><code>LQMS_OIDC_CLIENT_SECRET</code></td> <td><code>lqms-app-secret</code></td> <td>the confidential <code>lqms-app</code> client — <strong>in the realm import and in the app</strong>, from one variable</td> </tr> <tr> <td><code>LQMS_KC_ADMIN_API_SECRET</code></td> <td><code>lqms-admin-api-secret</code></td> <td>the confidential <code>lqms-admin-api</code> <strong>service-account</strong> client (§5.3) — again in the realm import and in the app, from one variable</td> </tr> <tr> <td><code>LQMS_ADMIN_PASSWORD</code></td> <td><code>demo</code> / <code>demo</code></td> <td>the first LQMS administrator's realm password</td> </tr> <tr> <td><code>LQMS_ADMIN_USERNAME</code> / <code>_EMAIL</code> / <code>_FIRST_NAME</code> / <code>_LAST_NAME</code></td> <td>the <code>demo</code> user's identity</td> <td>that same user — a real person instead of "Demo Administrator"</td> </tr> <tr> <td><code>LQMS_ORG_NAME</code></td> <td>the placeholder tenant "Operating Organization"</td> <td>the name the first boot founds <strong>your organization</strong> under (register 2026-08-06 #34). Not a secret — the interview asks for it. First bootstrap only; an installation that already booted renames through the governed API instead (§6.2)</td> </tr> </tbody> </table> <p>There is <strong>no row for demo story users</strong>, and that is the point: the realm template carries no demo people at all (feedback register 2026-08-06 #24, decided 2026-08-09). <code>anna.sommer</code>, <code>ben.keller</code>, <code>carla.fischer</code> and <code>dora.winter</code> belong to the localhost demo world and are created there by <code>deploy/seed-demo.sh</code>; nothing imports them here, so there is nothing to disable, nothing to rotate and no dormant account for an auditor to ask about.</p> <p>Generate with <code>openssl rand -hex 32</code> — 256 bits, and an alphabet with no shell metacharacters, which matters because these values pass through a compose env file, a container environment and (for the database role) a psql script.</p> <p>Not secrets, but required: <code>LQMS_DOMAIN</code>, <code>LQMS_ACME_EMAIL</code> and <code>LQMS_ORG_NAME</code> (in the table above because it, too, replaces a demo default).</p> <h3 id="52-how-the-realm-gets-them-vardefault-substitution">5.2 How the realm gets them: <code>${VAR:default}</code> substitution<a class="headerlink" href="#52-how-the-realm-gets-them-vardefault-substitution" title="Permanent link">¶</a></h3> <p>Keycloak's realm import resolves <code>${VAR}</code> and <code>${VAR:default}</code> placeholders from the process environment. Verified empirically on 26.4:</p> <ul> <li><code>${VAR:default}</code> → the environment value when set, the default when not. <code>deploy/lqms-realm.json</code> uses this form throughout, with <strong>the demo values as defaults</strong> — so the localhost stack imports exactly as before, and only a server that sets the variables gets different content.</li> <li><code>$(env:VAR)</code> and <code>${env.VAR}</code> are <strong>not</strong> substituted (they import as literal text).</li> <li><code>${VAR}</code> with no default and no environment value imports the <strong>literal string</strong> <code>${VAR}</code> — no error, no warning. That is why every one of these variables is <code>:?</code>-required in <code>docker-compose.server.yml</code>: a typo becomes a start-up failure instead of a client whose secret is the eleven characters <code>${LQMS_…}</code>.</li> <li>Quoted booleans (<code>"temporary": "${LQMS_ADMIN_PASSWORD_TEMPORARY:false}"</code>) are coerced correctly.</li> </ul> <p>What the server sets, beyond the credentials: <code>sslRequired=external</code> (the demo realm ships <code>none</code>), <code>redirectUris=https://<domain>/*</code> and <code>webOrigins=https://<domain></code> — closing the demo realm's wildcard redirect surface, which on a public confidential client is a real finding, not cosmetics.</p> <blockquote> <p><strong>The realm import only ever applies to a realm that does not exist yet.</strong> Changing <code>server.env</code> after the first successful boot changes nothing in Keycloak — use §10 to rotate.</p> </blockquote> <h3 id="53-inviting-people-who-have-no-account-yet--mail-and-the-one-credential-that-mints-logins">5.3 Inviting people who have no account yet — mail, and the one credential that mints logins<a class="headerlink" href="#53-inviting-people-who-have-no-account-yet--mail-and-the-one-credential-that-mints-logins" title="Permanent link">¶</a></h3> <p>This is optional configuration for an optional capability, and it is worth understanding before you decide whether you need it.</p> <p><strong>Without any of it</strong>, the product runs completely: documents, lifecycle, audit, the in-app notification inbox, and <em>inviting a person who already has a login here</em> (they answer in their invitations list). What you cannot do is <strong>invite an email address</strong> — the path a customer's first user arrives through, for somebody this installation has never seen. That one act refuses with a 409 naming these variables; nothing else changes.</p> <p><strong>With it</strong>, one invitation walks the whole way:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-10-1" name="__codelineno-10-1" href="#__codelineno-10-1"></a> tenant admin invites alice@customer.example <a id="__codelineno-10-2" name="__codelineno-10-2" href="#__codelineno-10-2"></a> │ <a id="__codelineno-10-3" name="__codelineno-10-3" href="#__codelineno-10-3"></a> ├─► LQMS mails her: https://qms.example.com/invite/<token> (LQMS's mail server) <a id="__codelineno-10-4" name="__codelineno-10-4" href="#__codelineno-10-4"></a> │ the token is single-use, expires in 14 days, and is stored <a id="__codelineno-10-5" name="__codelineno-10-5" href="#__codelineno-10-5"></a> │ only as a SHA-256 — the database holds no usable link <a id="__codelineno-10-6" name="__codelineno-10-6" href="#__codelineno-10-6"></a> │ <a id="__codelineno-10-7" name="__codelineno-10-7" href="#__codelineno-10-7"></a> ├─► she opens it, presses "set up access" <a id="__codelineno-10-8" name="__codelineno-10-8" href="#__codelineno-10-8"></a> │ LQMS creates her Keycloak account with NO credential at all <a id="__codelineno-10-9" name="__codelineno-10-9" href="#__codelineno-10-9"></a> │ and asks Keycloak to invite her to set one <a id="__codelineno-10-10" name="__codelineno-10-10" href="#__codelineno-10-10"></a> │ <a id="__codelineno-10-11" name="__codelineno-10-11" href="#__codelineno-10-11"></a> ├─► KEYCLOAK mails her its own setup link (Keycloak's mail server) <a id="__codelineno-10-12" name="__codelineno-10-12" href="#__codelineno-10-12"></a> │ UPDATE_PASSWORD + VERIFY_EMAIL, inside Keycloak's own pages <a id="__codelineno-10-13" name="__codelineno-10-13" href="#__codelineno-10-13"></a> │ <a id="__codelineno-10-14" name="__codelineno-10-14" href="#__codelineno-10-14"></a> └─► she sets her password, verifies her address, logs in <a id="__codelineno-10-15" name="__codelineno-10-15" href="#__codelineno-10-15"></a> → LQMS binds her by verified email and her invitation completes </code></pre></div> <p><strong>LQMS is never in the password path.</strong> It creates an account with no credentials array, and the secret is created by the person inside Keycloak's own flow, over a token Keycloak minted. This is why there are <strong>two mail senders for one mail server</strong>: the credential link is a Keycloak action token, so only Keycloak can send it. <code>docker-compose.server.yml</code> feeds both halves from the same <code>LQMS_SMTP_*</code> variables precisely so they cannot drift.</p> <p>What to set in <code>server.env</code> (all optional; an empty <code>LQMS_SMTP_HOST</code> is "no mail"):</p> <table> <thead> <tr> <th>Variable</th> <th>Consumed by</th> <th>Note</th> </tr> </thead> <tbody> <tr> <td><code>LQMS_SMTP_HOST</code> / <code>_PORT</code></td> <td>both</td> <td>empty host = no mail, which is the default posture</td> </tr> <tr> <td><code>LQMS_SMTP_USERNAME</code> / <code>_PASSWORD</code></td> <td>both</td> <td></td> </tr> <tr> <td><code>LQMS_SMTP_FROM</code> / <code>_FROM_NAME</code></td> <td>both</td> <td>defaults to <code>lqms@<LQMS_DOMAIN></code>, which most relays refuse — set a mailbox you own</td> </tr> <tr> <td><code>LQMS_SMTP_AUTH</code> / <code>_STARTTLS</code> / <code>_SSL</code></td> <td>Keycloak's half</td> <td>booleans</td> </tr> <tr> <td><code>LQMS_SMTP_START_TLS</code></td> <td>the app's half</td> <td><code>DISABLED</code> / <code>OPTIONAL</code> / <code>REQUIRED</code> — the same decision in quarkus-mailer's spelling</td> </tr> <tr> <td><code>LQMS_KC_ADMIN_API_SECRET</code></td> <td>both</td> <td><strong>required</strong> (§5.1) — the service-account credential that mints accounts</td> </tr> <tr> <td><code>LQMS_NOTIFICATIONS_EMAIL_ENABLED</code></td> <td>the app</td> <td>off even when mail works; turn it on after the relay is proven</td> </tr> </tbody> </table> <p><code>LQMS_APP_BASE_URL</code> is derived from <code>LQMS_DOMAIN</code> in the compose file and needs no row. It is configuration and <strong>never the request's <code>Host</code> header</strong> — a link is built in one request and followed from a mail client in another, and trusting <code>Host</code> is exactly how password-reset poisoning works.</p> <h4 id="existing-installations-the-realm-was-imported-once-so-add-the-client-by-hand">Existing installations: the realm was imported once, so add the client by hand<a class="headerlink" href="#existing-installations-the-realm-was-imported-once-so-add-the-client-by-hand" title="Permanent link">¶</a></h4> <p><code>smtpServer</code> and the <code>lqms-admin-api</code> client are part of the realm template, and <strong>the realm import only ever applies to a realm that does not exist yet</strong> (§5.2). A server that was deployed before this feature therefore needs both added to its live realm. Through the tunnel (§8b), with <code>kcadm</code> already configured:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-11-1" name="__codelineno-11-1" href="#__codelineno-11-1"></a><span class="nv">K</span><span class="o">=</span>/opt/keycloak/bin/kcadm.sh <a id="__codelineno-11-2" name="__codelineno-11-2" href="#__codelineno-11-2"></a><span class="nv">C</span><span class="o">=</span><span class="s1">'./lqms-compose exec keycloak'</span> <a id="__codelineno-11-3" name="__codelineno-11-3" href="#__codelineno-11-3"></a> <a id="__codelineno-11-4" name="__codelineno-11-4" href="#__codelineno-11-4"></a><span class="c1"># 0. authenticate kcadm (once per shell)</span> <a id="__codelineno-11-5" name="__codelineno-11-5" href="#__codelineno-11-5"></a><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>config<span class="w"> </span>credentials<span class="w"> </span>--server<span class="w"> </span>http://localhost:8080<span class="w"> </span>--realm<span class="w"> </span>master<span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-6" name="__codelineno-11-6" href="#__codelineno-11-6"></a><span class="w"> </span>--user<span class="w"> </span>admin<span class="w"> </span>--password<span class="w"> </span><span class="s2">"</span><span class="nv">$LQMS_KC_ADMIN_PASSWORD</span><span class="s2">"</span> <a id="__codelineno-11-7" name="__codelineno-11-7" href="#__codelineno-11-7"></a> <a id="__codelineno-11-8" name="__codelineno-11-8" href="#__codelineno-11-8"></a><span class="c1"># 1. the realm's own mail server — Keycloak sends the credential link itself</span> <a id="__codelineno-11-9" name="__codelineno-11-9" href="#__codelineno-11-9"></a><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>update<span class="w"> </span>realms/lqms<span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-10" name="__codelineno-11-10" href="#__codelineno-11-10"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.host=smtp.example.com'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-11" name="__codelineno-11-11" href="#__codelineno-11-11"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.port=587'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-12" name="__codelineno-11-12" href="#__codelineno-11-12"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.from=qms@example.com'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-13" name="__codelineno-11-13" href="#__codelineno-11-13"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.fromDisplayName=LQMS'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-14" name="__codelineno-11-14" href="#__codelineno-11-14"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.auth=true'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-15" name="__codelineno-11-15" href="#__codelineno-11-15"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.starttls=true'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-16" name="__codelineno-11-16" href="#__codelineno-11-16"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.user=<smtp user>'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-17" name="__codelineno-11-17" href="#__codelineno-11-17"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'smtpServer.password=<smtp password>'</span> <a id="__codelineno-11-18" name="__codelineno-11-18" href="#__codelineno-11-18"></a> <a id="__codelineno-11-19" name="__codelineno-11-19" href="#__codelineno-11-19"></a><span class="c1"># 2. the service-account client LQMS administers users with</span> <a id="__codelineno-11-20" name="__codelineno-11-20" href="#__codelineno-11-20"></a><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>create<span class="w"> </span>clients<span class="w"> </span>-r<span class="w"> </span>lqms<span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-21" name="__codelineno-11-21" href="#__codelineno-11-21"></a><span class="w"> </span>-s<span class="w"> </span><span class="nv">clientId</span><span class="o">=</span>lqms-admin-api<span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-22" name="__codelineno-11-22" href="#__codelineno-11-22"></a><span class="w"> </span>-s<span class="w"> </span><span class="s1">'name=LQMS identity administration'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-23" name="__codelineno-11-23" href="#__codelineno-11-23"></a><span class="w"> </span>-s<span class="w"> </span><span class="nv">enabled</span><span class="o">=</span><span class="nb">true</span><span class="w"> </span>-s<span class="w"> </span><span class="nv">publicClient</span><span class="o">=</span><span class="nb">false</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-24" name="__codelineno-11-24" href="#__codelineno-11-24"></a><span class="w"> </span>-s<span class="w"> </span><span class="nv">standardFlowEnabled</span><span class="o">=</span><span class="nb">false</span><span class="w"> </span>-s<span class="w"> </span><span class="nv">directAccessGrantsEnabled</span><span class="o">=</span><span class="nb">false</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-25" name="__codelineno-11-25" href="#__codelineno-11-25"></a><span class="w"> </span>-s<span class="w"> </span><span class="nv">serviceAccountsEnabled</span><span class="o">=</span><span class="nb">true</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-26" name="__codelineno-11-26" href="#__codelineno-11-26"></a><span class="w"> </span>-s<span class="w"> </span><span class="s2">"secret=</span><span class="nv">$LQMS_KC_ADMIN_API_SECRET</span><span class="s2">"</span> <a id="__codelineno-11-27" name="__codelineno-11-27" href="#__codelineno-11-27"></a> <a id="__codelineno-11-28" name="__codelineno-11-28" href="#__codelineno-11-28"></a><span class="c1"># 3. its ONE permission: realm-management -> manage-users, and nothing else</span> <a id="__codelineno-11-29" name="__codelineno-11-29" href="#__codelineno-11-29"></a><span class="nv">CID</span><span class="o">=</span><span class="k">$(</span><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>get<span class="w"> </span>clients<span class="w"> </span>-r<span class="w"> </span>lqms<span class="w"> </span>-q<span class="w"> </span><span class="nv">clientId</span><span class="o">=</span>lqms-admin-api<span class="w"> </span>--fields<span class="w"> </span>id<span class="w"> </span>--format<span class="w"> </span>csv<span class="w"> </span>--noquotes<span class="k">)</span> <a id="__codelineno-11-30" name="__codelineno-11-30" href="#__codelineno-11-30"></a><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>add-roles<span class="w"> </span>-r<span class="w"> </span>lqms<span class="w"> </span>--uusername<span class="w"> </span><span class="s2">"service-account-lqms-admin-api"</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-11-31" name="__codelineno-11-31" href="#__codelineno-11-31"></a><span class="w"> </span>--cclientid<span class="w"> </span>realm-management<span class="w"> </span>--rolename<span class="w"> </span>manage-users </code></pre></div> <p>Then verify, from the server, that the credential works and carries exactly that role:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-12-1" name="__codelineno-12-1" href="#__codelineno-12-1"></a><span class="c1"># a service-account token (never paste this into a shared log)</span> <a id="__codelineno-12-2" name="__codelineno-12-2" href="#__codelineno-12-2"></a>./lqms-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>keycloak<span class="w"> </span>sh<span class="w"> </span>-lc<span class="w"> </span><span class="s1">'curl -s -d grant_type=client_credentials \</span> <a id="__codelineno-12-3" name="__codelineno-12-3" href="#__codelineno-12-3"></a><span class="s1"> -d client_id=lqms-admin-api -d client_secret=$LQMS_KC_ADMIN_API_SECRET \</span> <a id="__codelineno-12-4" name="__codelineno-12-4" href="#__codelineno-12-4"></a><span class="s1"> http://localhost:8080/realms/lqms/protocol/openid-connect/token'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>head<span class="w"> </span>-c<span class="w"> </span><span class="m">40</span> <a id="__codelineno-12-5" name="__codelineno-12-5" href="#__codelineno-12-5"></a><span class="c1"># -> {"access_token":"eyJ... (anything else = the client or its secret is wrong)</span> </code></pre></div> <p>Finally, send one real invitation to an address you control and watch <strong>two</strong> messages arrive. One message means the other half's mail configuration is missing: LQMS's invitation but no Keycloak setup link is the realm's <code>smtpServer</code>; no invitation at all is the app's <code>quarkus.mailer.*</code>. The app's own refusal ("this installation cannot send mail") means <code>LQMS_SMTP_HOST</code> never reached the app container — check <code>./lqms-compose config</code> rather than the file.</p> <h4 id="existing-installations-credential-self-service-register-2026-08-06-27">Existing installations: credential self-service (register 2026-08-06 #27)<a class="headerlink" href="#existing-installations-credential-self-service-register-2026-08-06-27" title="Permanent link">¶</a></h4> <p><strong>Needed only on installs whose realm was imported before the credential-self-service change</strong> (fresh imports carry all of it). The realm template now enables self-service password reset ("Forgot Password?" on the sign-in page — it mails through the same <code>smtpServer</code> as above, so make sure mail works first), names the LQMS <strong>account theme</strong> for Keycloak's Account Console (which the app's person menu deep-links as "Change password"), and sets the realm display name the reset mail greets with ("your LQMS account", not "your Lqms account"). With <code>kcadm</code> authenticated as in step 0 above:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-13-1" name="__codelineno-13-1" href="#__codelineno-13-1"></a><span class="c1"># 4. self-service credentials: reset flag + console theme + mail display name (one call)</span> <a id="__codelineno-13-2" name="__codelineno-13-2" href="#__codelineno-13-2"></a><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>update<span class="w"> </span>realms/lqms<span class="w"> </span>-s<span class="w"> </span><span class="nv">resetPasswordAllowed</span><span class="o">=</span><span class="nb">true</span><span class="w"> </span>-s<span class="w"> </span><span class="nv">accountTheme</span><span class="o">=</span>lqms<span class="w"> </span>-s<span class="w"> </span><span class="nv">displayName</span><span class="o">=</span>LQMS <a id="__codelineno-13-3" name="__codelineno-13-3" href="#__codelineno-13-3"></a> <a id="__codelineno-13-4" name="__codelineno-13-4" href="#__codelineno-13-4"></a><span class="c1"># 5. the administrator imported with the ORIGINAL realm template holds no realm roles at all —</span> <a id="__codelineno-13-5" name="__codelineno-13-5" href="#__codelineno-13-5"></a><span class="c1"># without default-roles-lqms (account: manage-account/view-profile) the Account Console answers</span> <a id="__codelineno-13-6" name="__codelineno-13-6" href="#__codelineno-13-6"></a><span class="c1"># 401 and renders "Something went wrong" for exactly that user. Users minted later through the</span> <a id="__codelineno-13-7" name="__codelineno-13-7" href="#__codelineno-13-7"></a><span class="c1"># admin API (the ADR-0108 invite path) get the default role automatically; only realm-file</span> <a id="__codelineno-13-8" name="__codelineno-13-8" href="#__codelineno-13-8"></a><span class="c1"># imports skip it. Substitute your LQMS_ADMIN_USERNAME:</span> <a id="__codelineno-13-9" name="__codelineno-13-9" href="#__codelineno-13-9"></a><span class="nv">$C</span><span class="w"> </span><span class="nv">$K</span><span class="w"> </span>add-roles<span class="w"> </span>-r<span class="w"> </span>lqms<span class="w"> </span>--uusername<span class="w"> </span><span class="s2">"</span><span class="nv">$LQMS_ADMIN_USERNAME</span><span class="s2">"</span><span class="w"> </span>--rolename<span class="w"> </span>default-roles-lqms </code></pre></div> <p>The account theme itself ships as files (<code>deploy/keycloak-theme/lqms/account/</code>, mounted read-only by the compose file): make sure the deploy directory on the server is current (§3.4) and recreate the Keycloak container so the mount picks the directory up — the theme, unlike the realm settings above, needs no import and follows the files on every restart.</p> <hr /> <h2 id="6-bring-up">6. Bring-up<a class="headerlink" href="#6-bring-up" title="Permanent link">¶</a></h2> <div class="highlight"><pre><span></span><code><a id="__codelineno-14-1" name="__codelineno-14-1" href="#__codelineno-14-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-14-2" name="__codelineno-14-2" href="#__codelineno-14-2"></a> <a id="__codelineno-14-3" name="__codelineno-14-3" href="#__codelineno-14-3"></a><span class="c1"># 0. sanity: compose version + DNS</span> <a id="__codelineno-14-4" name="__codelineno-14-4" href="#__codelineno-14-4"></a>docker<span class="w"> </span>compose<span class="w"> </span>version<span class="w"> </span><span class="c1"># must be v2.24+</span> <a id="__codelineno-14-5" name="__codelineno-14-5" href="#__codelineno-14-5"></a>dig<span class="w"> </span>+short<span class="w"> </span><span class="s2">"</span><span class="nv">$DOMAIN</span><span class="s2">"</span><span class="w"> </span><span class="s2">"auth.</span><span class="nv">$DOMAIN</span><span class="s2">"</span> <a id="__codelineno-14-6" name="__codelineno-14-6" href="#__codelineno-14-6"></a> <a id="__codelineno-14-7" name="__codelineno-14-7" href="#__codelineno-14-7"></a><span class="c1"># 1. the registry credential (§4.2) — once per server, as the user that runs the stack</span> <a id="__codelineno-14-8" name="__codelineno-14-8" href="#__codelineno-14-8"></a><span class="nb">echo</span><span class="w"> </span><span class="s1">'<classic PAT with read:packages>'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>docker<span class="w"> </span>login<span class="w"> </span>ghcr.io<span class="w"> </span>-u<span class="w"> </span>nburri<span class="w"> </span>--password-stdin <a id="__codelineno-14-9" name="__codelineno-14-9" href="#__codelineno-14-9"></a> <a id="__codelineno-14-10" name="__codelineno-14-10" href="#__codelineno-14-10"></a><span class="c1"># 2. render the merged configuration and READ IT — no ports on app, 80/443 on caddy,</span> <a id="__codelineno-14-11" name="__codelineno-14-11" href="#__codelineno-14-11"></a><span class="c1"># 127.0.0.1:8081 on keycloak, nothing on postgres, and the app's `image:` naming the</span> <a id="__codelineno-14-12" name="__codelineno-14-12" href="#__codelineno-14-12"></a><span class="c1"># LQMS_VERSION you pinned (there must be no `build:` section at all)</span> <a id="__codelineno-14-13" name="__codelineno-14-13" href="#__codelineno-14-13"></a>docker<span class="w"> </span>compose<span class="w"> </span>--env-file<span class="w"> </span>server.env<span class="w"> </span><span class="se">\</span> <a id="__codelineno-14-14" name="__codelineno-14-14" href="#__codelineno-14-14"></a><span class="w"> </span>-f<span class="w"> </span>docker-compose.yml<span class="w"> </span>-f<span class="w"> </span>docker-compose.server.yml<span class="w"> </span>-p<span class="w"> </span>lqms<span class="w"> </span>config<span class="w"> </span><span class="p">|</span><span class="w"> </span>less <a id="__codelineno-14-15" name="__codelineno-14-15" href="#__codelineno-14-15"></a> <a id="__codelineno-14-16" name="__codelineno-14-16" href="#__codelineno-14-16"></a><span class="c1"># 3. pull the pinned release. Never `--build`: there is no build context on this box</span> <a id="__codelineno-14-17" name="__codelineno-14-17" href="#__codelineno-14-17"></a>docker<span class="w"> </span>compose<span class="w"> </span>--env-file<span class="w"> </span>server.env<span class="w"> </span><span class="se">\</span> <a id="__codelineno-14-18" name="__codelineno-14-18" href="#__codelineno-14-18"></a><span class="w"> </span>-f<span class="w"> </span>docker-compose.yml<span class="w"> </span>-f<span class="w"> </span>docker-compose.server.yml<span class="w"> </span>-p<span class="w"> </span>lqms<span class="w"> </span>pull <a id="__codelineno-14-19" name="__codelineno-14-19" href="#__codelineno-14-19"></a> <a id="__codelineno-14-20" name="__codelineno-14-20" href="#__codelineno-14-20"></a><span class="c1"># 4. start</span> <a id="__codelineno-14-21" name="__codelineno-14-21" href="#__codelineno-14-21"></a>docker<span class="w"> </span>compose<span class="w"> </span>--env-file<span class="w"> </span>server.env<span class="w"> </span><span class="se">\</span> <a id="__codelineno-14-22" name="__codelineno-14-22" href="#__codelineno-14-22"></a><span class="w"> </span>-f<span class="w"> </span>docker-compose.yml<span class="w"> </span>-f<span class="w"> </span>docker-compose.server.yml<span class="w"> </span>-p<span class="w"> </span>lqms<span class="w"> </span>up<span class="w"> </span>-d <a id="__codelineno-14-23" name="__codelineno-14-23" href="#__codelineno-14-23"></a> <a id="__codelineno-14-24" name="__codelineno-14-24" href="#__codelineno-14-24"></a><span class="c1"># 5. watch the first boot — certificates, realm import, migrations, bootstrap</span> <a id="__codelineno-14-25" name="__codelineno-14-25" href="#__codelineno-14-25"></a>docker<span class="w"> </span>compose<span class="w"> </span>-p<span class="w"> </span>lqms<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span>caddy<span class="w"> </span>keycloak<span class="w"> </span>app </code></pre></div> <p>If step 3 fails with <code>denied</code> or <code>unauthorized</code>, the credential is the problem, not the tag: re-read §4.2 (classic token, <code>read:packages</code>, and the login must belong to the <em>same user</em> that runs compose). If it fails with <code>manifest unknown</code>, <code>LQMS_VERSION</code> names a tag that was never published — check <a href="https://github.com/nburri/lqms/pkgs/container/lqms">https://github.com/nburri/lqms/pkgs/container/lqms</a>.</p> <p>Because the full command is long and every operation must use <strong>both</strong> files, define it once:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-15-1" name="__codelineno-15-1" href="#__codelineno-15-1"></a><span class="c1"># /home/lqms-ops/lqms/deploy/lqms-compose (chmod +x) — or a shell alias</span> <a id="__codelineno-15-2" name="__codelineno-15-2" href="#__codelineno-15-2"></a><span class="c1">#!/bin/sh</span> <a id="__codelineno-15-3" name="__codelineno-15-3" href="#__codelineno-15-3"></a><span class="nb">exec</span><span class="w"> </span>docker<span class="w"> </span>compose<span class="w"> </span>--env-file<span class="w"> </span>/home/lqms-ops/lqms/deploy/server.env<span class="w"> </span><span class="se">\</span> <a id="__codelineno-15-4" name="__codelineno-15-4" href="#__codelineno-15-4"></a><span class="w"> </span>-f<span class="w"> </span>/home/lqms-ops/lqms/deploy/docker-compose.yml<span class="w"> </span><span class="se">\</span> <a id="__codelineno-15-5" name="__codelineno-15-5" href="#__codelineno-15-5"></a><span class="w"> </span>-f<span class="w"> </span>/home/lqms-ops/lqms/deploy/docker-compose.server.yml<span class="w"> </span>-p<span class="w"> </span>lqms<span class="w"> </span><span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span> </code></pre></div> <p>Running plain <code>docker compose up -d</code> in <code>deploy/</code> (base file only) on a server would recreate the containers with <strong>demo credentials and published ports</strong>. Use the wrapper.</p> <p>What a healthy first boot looks like, in order:</p> <ol> <li><strong>caddy</strong> — <code>certificate obtained successfully</code> twice (once per name). Failures here are DNS or port 80 reachability, not TLS.</li> <li><strong>keycloak</strong> — <code>Realm 'lqms' imported</code>, <code>Import finished successfully</code>, then <code>Keycloak 26.4.x … started</code>, <code>Profile prod activated</code>.</li> <li><strong>app</strong> — Flyway applying migrations, then <code>Deployment checks passed (fan-out DEFINER owner can bypass RLS.)</code>, <code>System bootstrap complete for admin subject '11111111-…'</code>, then the Quarkus start line.</li> <li><code>docker compose -p lqms ps</code> shows all four services <code>healthy</code>.</li> </ol> <p>Then run the <strong>first-deploy checklist</strong> in §12 — it is short, and it covers exactly the things that cannot be verified without a real domain.</p> <h3 id="61-first-content-wizard-birth-or-demo-seed">6.1 First content: wizard birth or demo seed<a class="headerlink" href="#61-first-content-wizard-birth-or-demo-seed" title="Permanent link">¶</a></h3> <ul> <li><strong>Real installation → the wizard.</strong> Log in at <code>https://<domain>/</code> as the administrator from <code>server.env</code>. The app bootstraps that subject as system administrator on every boot (ADR-0012), and your organization already exists — the first boot founded it under <code>LQMS_ORG_NAME</code> (register 2026-08-06 #34). The setup wizard takes it from there: the project, the scopes, the document types. Nothing else is required — an empty start is a supported state.</li> <li><strong>Evaluation installation → the demo corpus.</strong> <code>./seed-demo.sh</code> is on the box (§3.4 syncs it), but it authenticates with the <strong>demo client secret, the demo Keycloak master password and demo user passwords</strong>, all of which a server deployment has replaced — and its first act is to <strong>create four story accounts</strong> (<code>anna.sommer</code>, <code>ben.keller</code>, <code>carla.fischer</code>, <code>dora.winter</code>) in the realm, which is exactly what this installation is built not to have. Seed on the localhost stack, not on the server.</li> </ul> <h3 id="62-existing-installations-give-the-tenant-its-real-name">6.2 Existing installations: give the tenant its real name<a class="headerlink" href="#62-existing-installations-give-the-tenant-its-real-name" title="Permanent link">¶</a></h3> <p>An installation whose first boot predates <code>LQMS_ORG_NAME</code> (or ran with it unset) carries its organization under the bootstrap placeholder <strong>"Operating Organization"</strong>. Setting the variable now changes nothing, and that is by design: the bootstrap resolves the tenant through the <code>ORG</code> scope's immutable code and <strong>never renames</strong> — a tenant's name moves only through a governed, audited act (<code>MANDATOR_RENAMED</code> in the GLOBAL trail), never as a configuration side effect of a boot.</p> <p>The rename lives on the provisioning API (<code>PUT /api/admin/provisioning/mandators/{id}/name</code>, MANAGE_MANDATORS-gated — the bootstrap administrator holds it). Once the admin UI grows the affordance you can do this from the Scopes page; until then, two calls:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-16-1" name="__codelineno-16-1" href="#__codelineno-16-1"></a><span class="c1"># A direct-grant token for the administrator — the same pattern as the §12 token check. server.env</span> <a id="__codelineno-16-2" name="__codelineno-16-2" href="#__codelineno-16-2"></a><span class="c1"># is a COMPOSE env file, not a shell script (values are unquoted and may contain spaces since</span> <a id="__codelineno-16-3" name="__codelineno-16-3" href="#__codelineno-16-3"></a><span class="c1"># LQMS_ORG_NAME), so read the rows out rather than sourcing it. If the admin password was changed</span> <a id="__codelineno-16-4" name="__codelineno-16-4" href="#__codelineno-16-4"></a><span class="c1"># at first login (LQMS_ADMIN_PASSWORD_TEMPORARY=true), use the current one, not server.env's.</span> <a id="__codelineno-16-5" name="__codelineno-16-5" href="#__codelineno-16-5"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-16-6" name="__codelineno-16-6" href="#__codelineno-16-6"></a><span class="nv">LQMS_DOMAIN</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>sed<span class="w"> </span>-n<span class="w"> </span><span class="s1">'s/^LQMS_DOMAIN=//p'</span><span class="w"> </span>server.env<span class="k">)</span><span class="s2">"</span> <a id="__codelineno-16-7" name="__codelineno-16-7" href="#__codelineno-16-7"></a><span class="nv">LQMS_OIDC_CLIENT_SECRET</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>sed<span class="w"> </span>-n<span class="w"> </span><span class="s1">'s/^LQMS_OIDC_CLIENT_SECRET=//p'</span><span class="w"> </span>server.env<span class="k">)</span><span class="s2">"</span> <a id="__codelineno-16-8" name="__codelineno-16-8" href="#__codelineno-16-8"></a><span class="nv">LQMS_ADMIN_USERNAME</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>sed<span class="w"> </span>-n<span class="w"> </span><span class="s1">'s/^LQMS_ADMIN_USERNAME=//p'</span><span class="w"> </span>server.env<span class="k">)</span><span class="s2">"</span> <a id="__codelineno-16-9" name="__codelineno-16-9" href="#__codelineno-16-9"></a><span class="nv">LQMS_ADMIN_PASSWORD</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>sed<span class="w"> </span>-n<span class="w"> </span><span class="s1">'s/^LQMS_ADMIN_PASSWORD=//p'</span><span class="w"> </span>server.env<span class="k">)</span><span class="s2">"</span> <a id="__codelineno-16-10" name="__codelineno-16-10" href="#__codelineno-16-10"></a><span class="nv">TOKEN</span><span class="o">=</span><span class="s2">"</span><span class="k">$(</span>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">"https://auth.</span><span class="nv">$LQMS_DOMAIN</span><span class="s2">/realms/lqms/protocol/openid-connect/token"</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-11" name="__codelineno-16-11" href="#__codelineno-16-11"></a><span class="w"> </span>-d<span class="w"> </span><span class="nv">grant_type</span><span class="o">=</span>password<span class="w"> </span>-d<span class="w"> </span><span class="nv">client_id</span><span class="o">=</span>lqms-app<span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-12" name="__codelineno-16-12" href="#__codelineno-16-12"></a><span class="w"> </span>--data-urlencode<span class="w"> </span><span class="s2">"client_secret=</span><span class="nv">$LQMS_OIDC_CLIENT_SECRET</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-13" name="__codelineno-16-13" href="#__codelineno-16-13"></a><span class="w"> </span>--data-urlencode<span class="w"> </span><span class="s2">"username=</span><span class="nv">$LQMS_ADMIN_USERNAME</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-14" name="__codelineno-16-14" href="#__codelineno-16-14"></a><span class="w"> </span>--data-urlencode<span class="w"> </span><span class="s2">"password=</span><span class="nv">$LQMS_ADMIN_PASSWORD</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-15" name="__codelineno-16-15" href="#__codelineno-16-15"></a><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-n<span class="w"> </span><span class="s1">'s/.*"access_token":"\([^"]*\)".*/\1/p'</span><span class="k">)</span><span class="s2">"</span> <a id="__codelineno-16-16" name="__codelineno-16-16" href="#__codelineno-16-16"></a> <a id="__codelineno-16-17" name="__codelineno-16-17" href="#__codelineno-16-17"></a><span class="c1"># 1. the tenant's id — the internal organization in the provisioning directory</span> <a id="__codelineno-16-18" name="__codelineno-16-18" href="#__codelineno-16-18"></a>curl<span class="w"> </span>-s<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer </span><span class="nv">$TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="s2">"https://</span><span class="nv">$LQMS_DOMAIN</span><span class="s2">/api/admin/provisioning/mandators"</span> <a id="__codelineno-16-19" name="__codelineno-16-19" href="#__codelineno-16-19"></a><span class="c1"># -> [{"id":"<uuid>","name":"Operating Organization","internal":true, …}]</span> <a id="__codelineno-16-20" name="__codelineno-16-20" href="#__codelineno-16-20"></a> <a id="__codelineno-16-21" name="__codelineno-16-21" href="#__codelineno-16-21"></a><span class="c1"># 2. the governed rename — 204. MANDATOR_RENAMED lands in the GLOBAL audit trail with the</span> <a id="__codelineno-16-22" name="__codelineno-16-22" href="#__codelineno-16-22"></a><span class="c1"># before/after names, and the tenant's ORG anchor scope follows while it is still named</span> <a id="__codelineno-16-23" name="__codelineno-16-23" href="#__codelineno-16-23"></a><span class="c1"># after the tenant (a deliberately relabelled anchor stays as it is).</span> <a id="__codelineno-16-24" name="__codelineno-16-24" href="#__codelineno-16-24"></a>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>PUT<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer </span><span class="nv">$TOKEN</span><span class="s2">"</span><span class="w"> </span>-H<span class="w"> </span><span class="s1">'Content-Type: application/json'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-25" name="__codelineno-16-25" href="#__codelineno-16-25"></a><span class="w"> </span>-d<span class="w"> </span><span class="s1">'{"name":"Your Organization AG"}'</span><span class="w"> </span><span class="se">\</span> <a id="__codelineno-16-26" name="__codelineno-16-26" href="#__codelineno-16-26"></a><span class="w"> </span><span class="s2">"https://</span><span class="nv">$LQMS_DOMAIN</span><span class="s2">/api/admin/provisioning/mandators/<uuid>/name"</span> </code></pre></div> <p>A name another organization already carries is a named <code>409</code> (two tenants must not share a name), a blank one a <code>400</code> — nothing moves on any refusal, and replaying the stored name writes no event.</p> <hr /> <h2 id="7-do-not-run-the-demo-scripts-against-a-server-stack">7. Do not run the demo scripts against a server stack<a class="headerlink" href="#7-do-not-run-the-demo-scripts-against-a-server-stack" title="Permanent link">¶</a></h2> <p><code>deploy/</code> ships scripts written for the localhost stack. Two of them are destructive here, because they hardcode <code>-f docker-compose.yml</code> (base only) and act on project <code>lqms</code>. The §3.4 sync keeps <code>smoke.sh</code> and <code>nightly-smoke.sh</code> <strong>off the box entirely</strong>; <code>seed-demo.sh</code> and <code>DEMO.md</code> do ride along with every sync and <strong>are present on the reference installation</strong>. Presence is not permission: none of these may be RUN against a server stack, and the table is what each one would do — whether it is already there or somebody copies it over:</p> <table> <thead> <tr> <th>Script</th> <th>On a server</th> </tr> </thead> <tbody> <tr> <td><code>smoke.sh</code></td> <td><strong>DESTRUCTIVE — never run it on the server.</strong> Its cleanup step is <code>docker compose -p lqms down -v</code>, which deletes the database, the content blobs and the Keycloak volume. It also rebuilds the image.</td> </tr> <tr> <td><code>nightly-smoke.sh</code></td> <td>Builds at HEAD and runs an isolated project (<code>lqms-nightly</code>, ports 8180/8181). Not destructive to <code>lqms</code>, but it publishes ports (see the ufw note in §3.2) and competes for RAM. Run it on the build machine.</td> </tr> <tr> <td><code>seed-demo.sh</code></td> <td><strong>Pollutes this installation.</strong> Beyond writing a synthetic mandator, scopes and documents into your database, its first step <strong>creates four demo accounts</strong> (<code>anna.sommer</code>, <code>ben.keller</code>, <code>carla.fischer</code>, <code>dora.winter</code>) in your Keycloak realm through the admin API — the very thing the realm template stopped shipping. It also assumes the demo secrets, so on a properly configured server it fails early rather than half-way; do not "fix" it by supplying real ones.</td> </tr> <tr> <td><code>restore.sh</code></td> <td>Legitimate, but it recreates the stack from the <strong>base</strong> file — see §9.4 for the correct sequence.</td> </tr> <tr> <td><code>verify-backup.sh</code></td> <td>Legitimate; prefer running it elsewhere (§9.3).</td> </tr> <tr> <td><code>backup.sh</code></td> <td><strong>Safe.</strong> It only uses <code>exec</code>, <code>stop</code>, <code>start</code> and <code>ps</code>, so it never recreates a container with base-file configuration.</td> </tr> </tbody> </table> <h3 id="71-base-file-only-scripts-and-the-pulled-image">7.1 Base-file-only scripts and the pulled image<a class="headerlink" href="#71-base-file-only-scripts-and-the-pulled-image" title="Permanent link">¶</a></h3> <p>There is a second, quieter consequence of <code>-f docker-compose.yml</code> on a pull-based server. The base file names <code>image: ${LQMS_IMAGE:-lqms:latest}</code> and still carries a <code>build:</code> section (the server override deletes it, but these scripts do not load the override). On this box <code>lqms:latest</code> does not exist — so <code>docker compose up</code> would try to <strong>build</strong> it, against a repository root that is not there.</p> <p>Give those scripts the image explicitly, derived from the same variable the deployment is pinned to so the two cannot drift:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-17-1" name="__codelineno-17-1" href="#__codelineno-17-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-17-2" name="__codelineno-17-2" href="#__codelineno-17-2"></a><span class="nb">set</span><span class="w"> </span>-a<span class="p">;</span><span class="w"> </span>.<span class="w"> </span>./server.env<span class="p">;</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span>+a <a id="__codelineno-17-3" name="__codelineno-17-3" href="#__codelineno-17-3"></a><span class="nb">export</span><span class="w"> </span><span class="nv">LQMS_IMAGE</span><span class="o">=</span><span class="s2">"ghcr.io/nburri/lqms:</span><span class="nv">$LQMS_VERSION</span><span class="s2">"</span> </code></pre></div> <p>Do this before <code>restore.sh</code> (§9.4) or <code>verify-backup.sh</code> (§9.3). With the image present locally, compose uses it and never reaches for the build context.</p> <hr /> <h2 id="8-keycloak-admin-access">8. Keycloak admin access<a class="headerlink" href="#8-keycloak-admin-access" title="Permanent link">¶</a></h2> <p>The admin console is not on the public route. Two ways in, both requiring SSH to the box:</p> <p><strong>a) The console, through an SSH tunnel</strong> (for user administration, sessions, realm settings):</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-18-1" name="__codelineno-18-1" href="#__codelineno-18-1"></a><span class="c1"># from your workstation</span> <a id="__codelineno-18-2" name="__codelineno-18-2" href="#__codelineno-18-2"></a>ssh<span class="w"> </span>-L<span class="w"> </span><span class="m">8081</span>:localhost:8081<span class="w"> </span>you@server <a id="__codelineno-18-3" name="__codelineno-18-3" href="#__codelineno-18-3"></a><span class="c1"># then open http://localhost:8081/admin/ in your browser</span> </code></pre></div> <p>This works because Keycloak is published on <code>127.0.0.1:8081</code> (unreachable from the network) and <code>KC_HOSTNAME_ADMIN=http://localhost:8081</code> tells Keycloak to build admin-console URLs for that address instead of the public one. The realm's <code>sslRequired=external</code> still permits plain HTTP here: the request arrives from a private (Docker bridge) address, which "external" exempts.</p> <p><strong>b) <code>kcadm</code> inside the container</strong> (for scripted changes — no tunnel, no browser):</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-19-1" name="__codelineno-19-1" href="#__codelineno-19-1"></a>./lqms-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>keycloak<span class="w"> </span>/opt/keycloak/bin/kcadm.sh<span class="w"> </span>config<span class="w"> </span>credentials<span class="w"> </span><span class="se">\</span> <a id="__codelineno-19-2" name="__codelineno-19-2" href="#__codelineno-19-2"></a><span class="w"> </span>--server<span class="w"> </span>http://localhost:8080<span class="w"> </span>--realm<span class="w"> </span>master<span class="w"> </span>--user<span class="w"> </span>admin<span class="w"> </span>--password<span class="w"> </span><span class="s2">"</span><span class="nv">$LQMS_KC_ADMIN_PASSWORD</span><span class="s2">"</span> <a id="__codelineno-19-3" name="__codelineno-19-3" href="#__codelineno-19-3"></a>./lqms-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>keycloak<span class="w"> </span>/opt/keycloak/bin/kcadm.sh<span class="w"> </span>get<span class="w"> </span>realms/lqms<span class="w"> </span>--fields<span class="w"> </span>sslRequired </code></pre></div> <p><strong>Right after the first boot</strong>, do the two things Keycloak 26 expects of you:</p> <ol> <li>Create a <strong>named permanent admin</strong> in the master realm and delete the temporary bootstrap <code>admin</code> user (the log says <code>Created temporary admin user with username admin</code>).</li> <li>Confirm the LQMS realm's users are what you expect: <strong>your administrator, and nobody else</strong> (plus the hidden <code>service-account-lqms-admin-api</code>, which the user list does not show). The realm template carries no demo people — if <code>anna.sommer</code>, <code>ben.keller</code>, <code>carla.fischer</code> or <code>dora.winter</code> is there, something ran <code>seed-demo.sh</code> against this box (§7) and the accounts should be deleted.</li> </ol> <p><strong>Password hygiene</strong>: the SERVER deployment imports the realm with a password policy — <code>length(12) and notUsername and notEmail</code>, injected by <code>docker-compose.server.yml</code> as <code>LQMS_PASSWORD_POLICY</code> — which binds every password <em>change</em> from then on. Two consequences:</p> <ul> <li>The policy is deliberately <strong>not</strong> in the realm template itself: Keycloak <strong>validates plaintext imported credentials against the policy</strong> (observed on 26.4 — a fresh import of the demo realm with its demo passwords under <code>length(12)</code> aborts startup with <code>invalidPasswordMinLengthMessage</code>), so a policy baked into the template would brick every demo-stack boot. The demo realm boots policy-free (the placeholder's empty default); real deployments inject it, and their generated 64-hex <code>server.env</code> passwords always pass. A server whose realm predates the policy (imports happen on FIRST boot only) adds it once, in the tunneled console (Authentication → Policies → Password policy) or via <code>kcadm</code>: <code>kcadm.sh update realms/lqms -s 'passwordPolicy=length(12) and notUsername and notEmail'</code>.</li> <li>When creating a user (e.g. the first real person after the neutral bootstrap admin), set their initial password with <strong>Temporary = ON</strong>: Keycloak then forces a proper, policy-checked password change at first login. The same lever exists for the bootstrap admin itself as <code>LQMS_ADMIN_PASSWORD_TEMPORARY=true</code> in <code>server.env</code> — leave it <code>false</code> only if you intend to run the scripted first-deploy token check before the browser login (§ checklist).</li> </ul> <hr /> <h2 id="8b-importing-a-corpus--the-one-shot-rule-read-before-the-production-import">8b. Importing a corpus — the one-shot rule (read before the production import)<a class="headerlink" href="#8b-importing-a-corpus--the-one-shot-rule-read-before-the-production-import" title="Permanent link">¶</a></h2> <p>An arrival import (<code>scripts/import/load_arrival.py</code>, ADR-0113) can be undone with <code>--dissolve-finalized</code> <strong>only while the imported scope is still a pure copy of the source.</strong></p> <p><strong>The window closes at the first act by any user</strong> — one training acknowledgement, one comment, one local release or revocation, one approval, one issued dossier, one disposal. From that moment <code>lqms_working_record_evidence</code> (V106) refuses dissolution <strong>permanently</strong>, and the corpus cannot be re-imported in place.</p> <p>This is deliberate, and the reason matters more than the rule: once somebody works in an imported project, <strong>the source system stops being the authority of record.</strong> Those acknowledgements and comments exist only in LQMS. A re-import would silently destroy work that has no other home.</p> <p><strong>Practical consequences</strong></p> <table> <thead> <tr> <th>Situation</th> <th>What to do</th> </tr> </thead> <tbody> <tr> <td>The import looks wrong, nobody has worked in it yet</td> <td><code>--dissolve-finalized</code>, fix the bundle, run again. Check <code>--dissolve-preview</code> first: it reads the same oracle the dissolution enforces, so it cannot disagree with what actually happens.</td> </tr> <tr> <td>The import looks wrong, people have already worked in it</td> <td>Dissolution is refused. Retire the corpus through the <strong>governed disposal lifecycle</strong>, which leaves a trail. (Dissolution means "this never happened"; disposal means "this happened and was retired" — for a real corpus the latter is what you want anyway.)</td> </tr> <tr> <td>Test system, want a clean slate</td> <td><strong>Full reset + re-seed</strong>, not dissolve-then-import. A wipe gives a genuinely known starting state; dissolution leaves behind whatever it does not reach (global catalog rows, roles, users, and the trail of the dissolution itself).</td> </tr> </tbody> </table> <p><strong>Therefore: inspect the arrived corpus before granting anyone access to work in it.</strong> The pre-flight check is cheap; the window is narrower than it looks.</p> <p>The refusal, when it fires, names its evidence — e.g. <em>"'MERIDIAN-QMS' has 5 acknowledgements, 3 comments. From the first record made in it an imported project is a working record system of this installation…"</em> — so an operator learns <strong>why</strong>, not merely <strong>that</strong>.</p> <h2 id="8c-exporting-a-corpus--the-portability-bundle-adr-0125">8c. Exporting a corpus — the portability bundle (ADR-0125)<a class="headerlink" href="#8c-exporting-a-corpus--the-portability-bundle-adr-0125" title="Permanent link">¶</a></h2> <p>The mirror of §8b: a scope leaves this installation as the same bundle format an arrival import consumes (<code>docs/architecture/arrival-bundle-format.md</code>, format 1.2). One endpoint, one audited act — and one refusal: an ORGANIZATION anchor is not exportable (it holds no documents; export the projects beneath it), while a PROJECT and the GLOBAL base library both are:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-20-1" name="__codelineno-20-1" href="#__codelineno-20-1"></a><span class="c1"># EXPORT_PROJECTS at GLOBAL is the gate (V115 seeded it to every role holding IMPORT_PROJECTS).</span> <a id="__codelineno-20-2" name="__codelineno-20-2" href="#__codelineno-20-2"></a><span class="c1"># The call streams a zip; the EXPORTED event lands on the scope's trail with counts.</span> <a id="__codelineno-20-3" name="__codelineno-20-3" href="#__codelineno-20-3"></a>curl<span class="w"> </span>-sf<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Authorization: Bearer </span><span class="nv">$TOKEN</span><span class="s2">"</span><span class="w"> </span>-o<span class="w"> </span>meridianqms-bundle.zip<span class="w"> </span><span class="s2">"https://lqms.ch/api/scopes/<scope-id>/portability-export"</span> </code></pre></div> <p><strong>What travels, what stays — read this before promising anyone a migration.</strong> The bundle carries the released corpus and its provenance: all versions per the arrival floor (RELEASED and REVOKED with releaser and date, an in-review draft as a draft), folders, attachments, native content trees, within-scope trace links, and the catalog rows the corpus uses. It does <strong>not</strong> carry the working-record layer: <strong>no audit trail, no approvals, no acknowledgements, no comments, no periodic reviews, no tasks, no save-points. A migration LOSES these BY DESIGN</strong> — the source instance, or its final backup, remains the archive of record for them (ADR-0125 §3). The bundle says this itself: its <code>README.txt</code> and manifest carry the loss statement, and an arrival import refuses a bundle that lacks it.</p> <p><strong>Completeness is structural, not asserted.</strong> The writer emits <code>manifest.json</code> last; if anything fails mid-stream there is no manifest, no <code>EXPORTED</code> event, and a zip the importer refuses. An export you can finalize an import from is complete; there is no "probably complete".</p> <p><strong>Landing it elsewhere</strong>: the target runs the ordinary §8b arrival (<code>load_arrival.py</code>) against the bundle. Ids land <strong>verbatim</strong> — the bundle declares <code>idsAuthoritative</code>, the import batch adopts every id including numbering gaps, and the target's counter is seeded past them — so every <code>doc:</code> cross-reference keeps meaning what it meant. The target scope adopts the source's scope code (ids embed it; the import refuses a divergent mapping). After landing, the one-shot rule of §8b governs the copy exactly like any arrival.</p> <h2 id="9-backups">9. Backups<a class="headerlink" href="#9-backups" title="Permanent link">¶</a></h2> <p>The mechanism, the retention rules and the disposal interplay are ADR-0060 / ADR-0025 / ADR-0040 and are documented in <a href="https://github.com/nburri/lqms/blob/main/deploy/README.md#backup--restore"><code>deploy/README.md</code></a> and <a href="../../architecture/07_deployment_view/"><code>07_deployment_view.md</code></a> §7.1. What is server-specific:</p> <h3 id="91-the-schedule--what-the-reference-installation-actually-runs">9.1 The schedule — what the reference installation actually runs<a class="headerlink" href="#91-the-schedule--what-the-reference-installation-actually-runs" title="Permanent link">¶</a></h3> <p>A <strong>user crontab owned by <code>lqms-ops</code></strong> (<code>crontab -e</code> as that user), not a root <code>/etc/cron.d</code> drop-in. A user crontab has no user column, and the choice is not cosmetic: the stack, the deploy directory (§3.4) and the <code>docker login</code> (§4.2) all belong to that same unprivileged user, and the backup must run as the identity that can reach the Docker socket and the archive directory.</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-21-1" name="__codelineno-21-1" href="#__codelineno-21-1"></a># crontab -e as lqms-ops — nightly full backup at 02:15 <a id="__codelineno-21-2" name="__codelineno-21-2" href="#__codelineno-21-2"></a>15 2 * * * cd /home/lqms-ops/lqms/deploy && LQMS_BACKUP_TARGET=/home/lqms-ops/backup ./backup.sh >> /home/lqms-ops/lqms-backup.log 2>&1 </code></pre></div> <p><code>backup.sh</code> briefly stops Keycloak (its H2 file must be unlocked for the realm export) and always restarts it — expect a ~10 second login outage at 02:15. Blobs are copied first and re-hashed, then <code>pg_dump -Fc</code>, then the realm export including users.</p> <blockquote> <p><strong>Two gaps in the reference installation — stated, not papered over.</strong> Both were verified on the box during the v0.9.15 deploy (2026-08-19). Both fixes are <strong>host-side and Nicolas's to apply</strong>; no code change is involved, and this note stays until they are.</p> <ol> <li><strong>The off-box target is on the same disk.</strong> <code>LQMS_BACKUP_TARGET=/home/lqms-ops/backup</code> sits on the host's own filesystem, so §9.2's rule — <em>a directory on the same disk is not a backup</em> — is currently violated by this very installation. The archives themselves are real and restorable; what is missing is the <strong>separation</strong> ADR-0018 requires, and the loss of the host would take every copy with it. <em>Pending:</em> repoint <code>LQMS_BACKUP_TARGET</code> at one of §9.2's genuinely separate destinations (a second host over <code>scp</code>, or object storage via <code>rclone</code>), and run the exact cron command by hand once so the credential is proven from cron's environment, not from your shell.</li> <li><strong>The dead-man ping is documented but not armed.</strong> <code>LQMS_BACKUP_PING_URL</code> is unset in the crontab above, so nothing fires on success and — the part that matters — nothing is watching for silence. A backup that quietly stops running would go unnoticed until somebody needed it. §9.2 and §14 both describe the ping as part of this deployment's monitoring; on this box it is not. <em>Pending:</em> register a check with a ping service, add the variable to the cron line, and configure that service to <strong>alarm on silence</strong> — otherwise the mechanism is decoration (§9.2).</li> </ol> <p>Until both are done, §12's check 16 does not pass on this installation, and the honest statement of the backup posture is: <em>nightly archives are produced and verifiable, but they are not stored separately and their absence is not alarmed.</em></p> </blockquote> <h3 id="92-the-off-server-copy-is-the-point">9.2 The off-server copy is the point<a class="headerlink" href="#92-the-off-server-copy-is-the-point" title="Permanent link">¶</a></h3> <p><code>LQMS_BACKUP_TARGET</code> is what makes the archive survive the loss of the server (ADR-0018 — "stored separately from the primary system"). <strong>A directory on the same disk is not a backup.</strong> The rule stands — and the reference installation currently breaks it: its target is <code>/home/lqms-ops/backup</code>, on the host's own disk (§9.1, gap 1; host-side fix pending). Pick one:</p> <table> <thead> <tr> <th>Target</th> <th><code>LQMS_BACKUP_TARGET</code></th> <th><code>LQMS_BACKUP_TARGET_CMD</code></th> </tr> </thead> <tbody> <tr> <td>a mounted NAS / second volume</td> <td><code>/mnt/backup/lqms</code></td> <td><code>cp</code> (default)</td> </tr> <tr> <td>another host over SSH</td> <td><code>backup@nas:/srv/lqms/</code></td> <td><code>scp</code></td> </tr> <tr> <td>object storage (S3, B2, …)</td> <td><code>remote:lqms-backups/</code></td> <td><code>rclone copyto</code></td> </tr> </tbody> </table> <p>For <code>scp</code>/<code>rclone</code>, root's key or rclone config must exist on the server (cron runs without your shell environment) — test the exact cron command once by hand.</p> <p>Two more things to get right, because neither is automatic:</p> <ul> <li><strong>Retention on the destination.</strong> <code>backup.sh</code> prunes only the <em>local</em> copies past <code>LQMS_BACKUP_RETENTION_DAYS</code> (default 35). Pruning the off-server copies to the same window is yours to configure — and it is a data-protection statement, not housekeeping: disposed content persists in backups for at most the retention window (ADR-0025 / REQ-DPR-011).</li> <li><strong>The dead-man ping.</strong> <code>LQMS_BACKUP_PING_URL</code> fires only on success. The <em>absence</em> of the ping is the alert — so the ping service must be configured to alarm on silence, or the whole mechanism is decoration. On the reference installation the variable is <strong>unset</strong>, so there is no ping and nothing watching for its silence (§9.1, gap 2; host-side fix pending).</li> </ul> <h3 id="93-restore-rehearsal--preferably-not-on-this-box">9.3 Restore rehearsal — preferably not on this box<a class="headerlink" href="#93-restore-rehearsal--preferably-not-on-this-box" title="Permanent link">¶</a></h3> <p><code>verify-backup.sh</code> restores the newest archive into a scratch project alongside the running stack. On a 2-4 vCPU / 8 GB server that means a second PostgreSQL, a second Keycloak and a second JVM competing with production, plus two publicly-published ports (8090/8091 — Docker bypasses ufw, §3.2; binding them to loopback does not work, because the script's own health check calls <code>http://localhost:${PORT}/api/version</code>).</p> <p>Run the rehearsal <strong>on another machine against the off-server copy</strong> instead. That is also the stronger test: it proves the copy that would survive the server is restorable. Weekly, e.g.</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-22-1" name="__codelineno-22-1" href="#__codelineno-22-1"></a># crontab -e on the REHEARSAL machine — its own copy of deploy/, its own paths, not the server's <a id="__codelineno-22-2" name="__codelineno-22-2" href="#__codelineno-22-2"></a>30 3 * * 0 cd ~/lqms/deploy && LQMS_BACKUP_DIR=/mnt/backup/lqms ./verify-backup.sh >> ~/lqms-verify.log 2>&1 </code></pre></div> <p>If it must run on the server, add the <code>DOCKER-USER</code> drop rule from §3.2 first and accept the contention.</p> <h4 id="what-a-rehearsal-record-proves--and-why-the-pre-2026-08-19-rows-do-not">What a rehearsal record proves — and why the pre-2026-08-19 rows do not<a class="headerlink" href="#what-a-rehearsal-record-proves--and-why-the-pre-2026-08-19-rows-do-not" title="Permanent link">¶</a></h4> <p>Each run appends one line to <code>deploy/backup-verification.log</code> (that file is the server's own output and is gitignored; it is never synced from a laptop, §3.4):</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-23-1" name="__codelineno-23-1" href="#__codelineno-23-1"></a><iso> archive=<zip> project=<name> result=PASS|FAIL|ABORTED exit=<n> checks=[<name>=<R> ...] </code></pre></div> <ul> <li>Every check is measured against something <strong>outside the restored copy</strong> — the archive's own <code>manifest.json</code> (<code>blobs.count</code>, <code>app.git_sha</code>, <code>database.sha256</code>). A restored store is never graded against its own numbers.</li> <li><code>result=ABORTED</code> means the run <strong>died before it had a verdict</strong> (crash, <code>set -e</code>, Ctrl-C, docker daemon gone); <code>exit=</code> carries the process status. An aborted rehearsal proves nothing, and is never recorded as PASS.</li> <li><strong>A manifest whose <code>app.git_sha</code> is <code>"unknown"</code> FAILS the version check</strong>, deliberately: there is then nothing outside the restored copy to compare the running image against, and a comparison with no yardstick must not be recorded as proof. Every archive produced on the reference installation up to and including the v0.9.15 deploy carries <code>unknown</code>, for a single reason — <code>backup.sh</code> asked the app on the host's <code>localhost:8080</code>, a port the server override never publishes (<code>ports: !reset []</code> on the app service). Fixed on 2026-08-21: the script now asks the app <strong>container</strong> directly over the compose project, falls back to <code>LQMS_APP_URL</code>, and prints a loud <code>WARNING</code> when neither answers. Archives written before that fix cannot prove which image they came from, so a rehearsal against one of them records <code>version_sha=FAIL</code> for that reason alone; archives written after it carry the real short sha.</li> <li>A per-check <code>SKIP</code> means there was genuinely <strong>nothing to measure</strong> — no blobs to re-hash, no <code>content_part</code> rows to resolve, no <code>database.sha256</code> claimed (in-app-produced archives state <code>null</code>, ADR-0116). It is deliberately neither PASS nor FAIL: an empty instance must not be dressed up as proof.</li> </ul> <p><strong>Records dated before 2026-08-19 are not load-bearing evidence</strong> (fix-log decision 4, 2026-08-19). The verifier that wrote them could pass vacuously: its blob check walked the restored store and reported "clean over N blobs" without reading the manifest, so an EMPTY restore gave N=0, mismatch=0 and recorded <code>result=PASS</code>; its closure check passed when the <code>content_part</code> query returned nothing, including when the query had FAILED; and its teardown captured the process exit code and never used it, so a crashed or interrupted rehearsal still appended <code>result=PASS</code>. Reproduced on 2026-08-19 with identical input (a 28-blob archive restored into an empty store): old script <code>result=PASS</code>, fixed script <code>result=FAIL</code>. Those restores may all have been perfectly good — the point is only that the records do not PROVE it. The rows are kept verbatim, with a dated note appended in the file itself; <strong>the first record written after that note is the first load-bearing rehearsal evidence</strong>, and that is what IQ evidence item 7 should cite from now on.</p> <h3 id="94-restoring-onto-the-server--the-exact-sequence">9.4 Restoring onto the server — the exact sequence<a class="headerlink" href="#94-restoring-onto-the-server--the-exact-sequence" title="Permanent link">¶</a></h3> <p><code>restore.sh</code> knows only the base compose file, so a plain restore recreates the stack in <strong>demo configuration</strong> (published ports, <code>start-dev</code> Keycloak, demo DB-role password on the freshly initialised volume). Sequence that ends in the right state:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-24-1" name="__codelineno-24-1" href="#__codelineno-24-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-24-2" name="__codelineno-24-2" href="#__codelineno-24-2"></a><span class="nb">set</span><span class="w"> </span>-a<span class="p">;</span><span class="w"> </span>.<span class="w"> </span>./server.env<span class="p">;</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span>+a<span class="w"> </span><span class="c1"># so the restore's compose interpolation sees your values</span> <a id="__codelineno-24-3" name="__codelineno-24-3" href="#__codelineno-24-3"></a><span class="nb">export</span><span class="w"> </span><span class="nv">LQMS_IMAGE</span><span class="o">=</span><span class="s2">"ghcr.io/nburri/lqms:</span><span class="nv">$LQMS_VERSION</span><span class="s2">"</span><span class="w"> </span><span class="c1"># §7.1 — or compose tries to BUILD</span> <a id="__codelineno-24-4" name="__codelineno-24-4" href="#__codelineno-24-4"></a> <a id="__codelineno-24-5" name="__codelineno-24-5" href="#__codelineno-24-5"></a>./restore.sh<span class="w"> </span>backups/lqms-backup-<ts>.zip<span class="w"> </span>--project<span class="w"> </span>lqms<span class="w"> </span>--force <a id="__codelineno-24-6" name="__codelineno-24-6" href="#__codelineno-24-6"></a> <a id="__codelineno-24-7" name="__codelineno-24-7" href="#__codelineno-24-7"></a><span class="c1"># the recreated postgres volume ran postgres-init.sql WITHOUT the server file's</span> <a id="__codelineno-24-8" name="__codelineno-24-8" href="#__codelineno-24-8"></a><span class="c1"># LQMS_MIGRATION_PASSWORD, so put the real passwords back before the app connects</span> <a id="__codelineno-24-9" name="__codelineno-24-9" href="#__codelineno-24-9"></a>./lqms-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>-T<span class="w"> </span>-e<span class="w"> </span><span class="nv">NEWPW</span><span class="o">=</span><span class="s2">"</span><span class="nv">$LQMS_MIGRATION_PASSWORD</span><span class="s2">"</span><span class="w"> </span>postgres<span class="w"> </span><span class="se">\</span> <a id="__codelineno-24-10" name="__codelineno-24-10" href="#__codelineno-24-10"></a><span class="w"> </span>psql<span class="w"> </span>-U<span class="w"> </span>postgres<span class="w"> </span>-d<span class="w"> </span>lqms<span class="w"> </span>-v<span class="w"> </span><span class="nv">ON_ERROR_STOP</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="s"><<'SQL'</span> <a id="__codelineno-24-11" name="__codelineno-24-11" href="#__codelineno-24-11"></a><span class="s">\set newpw `echo "$NEWPW"`</span> <a id="__codelineno-24-12" name="__codelineno-24-12" href="#__codelineno-24-12"></a><span class="s">ALTER ROLE lqms_migration PASSWORD :'newpw';</span> <a id="__codelineno-24-13" name="__codelineno-24-13" href="#__codelineno-24-13"></a><span class="s">SQL</span> <a id="__codelineno-24-14" name="__codelineno-24-14" href="#__codelineno-24-14"></a> <a id="__codelineno-24-15" name="__codelineno-24-15" href="#__codelineno-24-15"></a><span class="c1"># bring the stack back to the SERVER configuration (Caddy, no published app port, prod Keycloak)</span> <a id="__codelineno-24-16" name="__codelineno-24-16" href="#__codelineno-24-16"></a>./lqms-compose<span class="w"> </span>up<span class="w"> </span>-d <a id="__codelineno-24-17" name="__codelineno-24-17" href="#__codelineno-24-17"></a> <a id="__codelineno-24-18" name="__codelineno-24-18" href="#__codelineno-24-18"></a><span class="c1"># the realm export carries NO client secret (ADR-0060) — re-apply it (§10)</span> </code></pre></div> <p>Then work through the obligations <code>restore.sh</code> prints — in particular <strong>replaying disposals</strong> recorded after the archive's timestamp (a restore resurrects disposed content, ADR-0040).</p> <p><strong>If <code>--force</code> fails, read the message before assuming the worst.</strong> Since 2026-08-19 the script unpacks and validates the archive <em>before</em> it wipes anything: it requires <code>manifest.json</code>, a non-empty <code>db.dump</code> whose table of contents <code>pg_restore --list</code> can read, a <code>db.dump</code> matching the manifest's <code>database.sha256</code> when the archive states one, and — when the manifest claims blobs — a content half that is actually present. Any of these failing aborts with <code>NOTHING was destroyed</code>, and the existing project is untouched: fix or re-fetch the archive and re-run. Only failures reported <em>after</em> the <code>--force: tearing down …</code> line have cost you the old volumes. (A blob count that differs from the manifest but is not zero only WARNS — partial content is still worth restoring in a disaster — and the next rehearsal records that discrepancy as a failure.)</p> <hr /> <h2 id="10-rotating-a-secret-after-first-boot">10. Rotating a secret after first boot<a class="headerlink" href="#10-rotating-a-secret-after-first-boot" title="Permanent link">¶</a></h2> <p>The realm import runs once; after that, credentials are changed in place.</p> <p><strong>OIDC client secret</strong> (also the fix after any restore):</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-25-1" name="__codelineno-25-1" href="#__codelineno-25-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-25-2" name="__codelineno-25-2" href="#__codelineno-25-2"></a><span class="nb">set</span><span class="w"> </span>-a<span class="p">;</span><span class="w"> </span>.<span class="w"> </span>./server.env<span class="p">;</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span>+a <a id="__codelineno-25-3" name="__codelineno-25-3" href="#__codelineno-25-3"></a><span class="nv">KC</span><span class="o">=(</span>./lqms-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>-T<span class="w"> </span>keycloak<span class="w"> </span>/opt/keycloak/bin/kcadm.sh<span class="o">)</span> <a id="__codelineno-25-4" name="__codelineno-25-4" href="#__codelineno-25-4"></a><span class="s2">"</span><span class="si">${</span><span class="nv">KC</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span>config<span class="w"> </span>credentials<span class="w"> </span>--server<span class="w"> </span>http://localhost:8080<span class="w"> </span>--realm<span class="w"> </span>master<span class="w"> </span><span class="se">\</span> <a id="__codelineno-25-5" name="__codelineno-25-5" href="#__codelineno-25-5"></a><span class="w"> </span>--user<span class="w"> </span>admin<span class="w"> </span>--password<span class="w"> </span><span class="s2">"</span><span class="nv">$LQMS_KC_ADMIN_PASSWORD</span><span class="s2">"</span> <a id="__codelineno-25-6" name="__codelineno-25-6" href="#__codelineno-25-6"></a><span class="nv">CID</span><span class="o">=</span><span class="k">$(</span><span class="s2">"</span><span class="si">${</span><span class="nv">KC</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span>get<span class="w"> </span>clients<span class="w"> </span>-r<span class="w"> </span>lqms<span class="w"> </span>-q<span class="w"> </span><span class="nv">clientId</span><span class="o">=</span>lqms-app<span class="w"> </span>--fields<span class="w"> </span>id<span class="w"> </span>--format<span class="w"> </span>csv<span class="w"> </span>--noquotes<span class="k">)</span> <a id="__codelineno-25-7" name="__codelineno-25-7" href="#__codelineno-25-7"></a><span class="s2">"</span><span class="si">${</span><span class="nv">KC</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span><span class="w"> </span>update<span class="w"> </span><span class="s2">"clients/</span><span class="nv">$CID</span><span class="s2">"</span><span class="w"> </span>-r<span class="w"> </span>lqms<span class="w"> </span>-s<span class="w"> </span><span class="s2">"secret=</span><span class="nv">$LQMS_OIDC_CLIENT_SECRET</span><span class="s2">"</span> <a id="__codelineno-25-8" name="__codelineno-25-8" href="#__codelineno-25-8"></a>./lqms-compose<span class="w"> </span>up<span class="w"> </span>-d<span class="w"> </span>app<span class="w"> </span><span class="c1"># the app picks up the new value from server.env</span> </code></pre></div> <p><strong>Database role password:</strong> update <code>server.env</code>, then (same form as §9.4 — the value travels in the container environment, never in a command line that <code>ps</code> or your shell history can see):</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-26-1" name="__codelineno-26-1" href="#__codelineno-26-1"></a><span class="nb">set</span><span class="w"> </span>-a<span class="p">;</span><span class="w"> </span>.<span class="w"> </span>./server.env<span class="p">;</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span>+a <a id="__codelineno-26-2" name="__codelineno-26-2" href="#__codelineno-26-2"></a>./lqms-compose<span class="w"> </span><span class="nb">exec</span><span class="w"> </span>-T<span class="w"> </span>-e<span class="w"> </span><span class="nv">NEWPW</span><span class="o">=</span><span class="s2">"</span><span class="nv">$LQMS_MIGRATION_PASSWORD</span><span class="s2">"</span><span class="w"> </span>postgres<span class="w"> </span><span class="se">\</span> <a id="__codelineno-26-3" name="__codelineno-26-3" href="#__codelineno-26-3"></a><span class="w"> </span>psql<span class="w"> </span>-U<span class="w"> </span>postgres<span class="w"> </span>-d<span class="w"> </span>lqms<span class="w"> </span>-v<span class="w"> </span><span class="nv">ON_ERROR_STOP</span><span class="o">=</span><span class="m">1</span><span class="w"> </span><span class="s"><<'SQL'</span> <a id="__codelineno-26-4" name="__codelineno-26-4" href="#__codelineno-26-4"></a><span class="s">\set newpw `echo "$NEWPW"`</span> <a id="__codelineno-26-5" name="__codelineno-26-5" href="#__codelineno-26-5"></a><span class="s">ALTER ROLE lqms_migration PASSWORD :'newpw';</span> <a id="__codelineno-26-6" name="__codelineno-26-6" href="#__codelineno-26-6"></a><span class="s">SQL</span> <a id="__codelineno-26-7" name="__codelineno-26-7" href="#__codelineno-26-7"></a>./lqms-compose<span class="w"> </span>up<span class="w"> </span>-d<span class="w"> </span>app<span class="w"> </span><span class="c1"># so the app reconnects with the new value</span> </code></pre></div> <p>The same shape rotates the superuser: <code>ALTER ROLE postgres PASSWORD :'newpw'</code> with <code>LQMS_POSTGRES_PASSWORD</code>.</p> <p><strong>Keycloak admin / user passwords:</strong> through the admin console (§8) — <code>KC_BOOTSTRAP_ADMIN_PASSWORD</code> only ever applies to the creation of the temporary admin user.</p> <p>Anything you change in <code>server.env</code> needs <code>./lqms-compose up -d</code> to reach a container; compose recreates only the services whose configuration actually changed.</p> <hr /> <h2 id="11-keeping-it-current">11. Keeping it current<a class="headerlink" href="#11-keeping-it-current" title="Permanent link">¶</a></h2> <table> <thead> <tr> <th>Layer</th> <th>How</th> <th>Cadence</th> </tr> </thead> <tbody> <tr> <td>OS packages</td> <td><code>unattended-upgrades</code> (§3.3)</td> <td>automatic</td> </tr> <tr> <td><strong>Container images</strong> — <code>postgres:18</code>, <code>quay.io/keycloak/keycloak:26.4</code>, <code>caddy:2-alpine</code></td> <td><code>./lqms-compose pull && ./lqms-compose up -d</code> — pulls the newest <strong>patch</strong> of each pinned tag</td> <td>monthly, and on a published CVE</td> </tr> <tr> <td>The LQMS image</td> <td>edit <code>LQMS_VERSION</code> in <code>server.env</code> → <code>./lqms-compose pull app</code> → <code>./lqms-compose up -d app</code> (§11.1)</td> <td>per release</td> </tr> </tbody> </table> <p>Two cautions:</p> <ul> <li><strong>Never move <code>postgres:18</code> to a new major</strong> (<code>postgres:19</code>) by editing the tag: PostgreSQL major upgrades require a dump/restore of the data directory, and the container will refuse to start on a data directory from the previous major. Treat it as a planned migration.</li> <li><strong>Take a backup before any upgrade</strong>, and check <code>/api/version</code> afterwards to confirm what is actually running (§12).</li> </ul> <h3 id="111-upgrading-lqms--the-whole-ritual">11.1 Upgrading LQMS — the whole ritual<a class="headerlink" href="#111-upgrading-lqms--the-whole-ritual" title="Permanent link">¶</a></h3> <p>An upgrade is <strong>one edited line and two commands</strong>. There is no build, no transfer, no repository.</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-27-1" name="__codelineno-27-1" href="#__codelineno-27-1"></a><span class="nb">cd</span><span class="w"> </span>/home/lqms-ops/lqms/deploy <a id="__codelineno-27-2" name="__codelineno-27-2" href="#__codelineno-27-2"></a> <a id="__codelineno-27-3" name="__codelineno-27-3" href="#__codelineno-27-3"></a>./backup.sh<span class="w"> </span><span class="c1"># first. Always. (§9)</span> <a id="__codelineno-27-4" name="__codelineno-27-4" href="#__codelineno-27-4"></a> <a id="__codelineno-27-5" name="__codelineno-27-5" href="#__codelineno-27-5"></a><span class="nv">$EDITOR</span><span class="w"> </span>server.env<span class="w"> </span><span class="c1"># LQMS_VERSION=v0.9.0 -> LQMS_VERSION=v0.10.0</span> <a id="__codelineno-27-6" name="__codelineno-27-6" href="#__codelineno-27-6"></a>./lqms-compose<span class="w"> </span>pull<span class="w"> </span>app<span class="w"> </span><span class="c1"># fetch exactly that tag from ghcr.io</span> <a id="__codelineno-27-7" name="__codelineno-27-7" href="#__codelineno-27-7"></a>./lqms-compose<span class="w"> </span>up<span class="w"> </span>-d<span class="w"> </span>app<span class="w"> </span><span class="c1"># recreate the app container on the new image</span> <a id="__codelineno-27-8" name="__codelineno-27-8" href="#__codelineno-27-8"></a> <a id="__codelineno-27-9" name="__codelineno-27-9" href="#__codelineno-27-9"></a>./lqms-compose<span class="w"> </span>logs<span class="w"> </span>-f<span class="w"> </span>app<span class="w"> </span><span class="c1"># watch Flyway migrate, then the Quarkus start line</span> <a id="__codelineno-27-10" name="__codelineno-27-10" href="#__codelineno-27-10"></a>curl<span class="w"> </span>-s<span class="w"> </span>https://<domain>/api/version<span class="w"> </span><span class="c1"># confirm productVersion is what you just pinned</span> </code></pre></div> <p>Notes that matter:</p> <ul> <li><strong>Migrations run on boot.</strong> The app applies its Flyway migrations at start-up, so the upgrade is the deploy. Expect the first start after an upgrade to take longer than a restart; watch the log rather than assuming.</li> <li><strong>Only <code>app</code> is touched.</strong> <code>pull app</code> / <code>up -d app</code> leave PostgreSQL, Keycloak and Caddy alone. A bare <code>./lqms-compose pull</code> also fetches new patches of those three — a separate decision (the table above), not something to do accidentally during an LQMS upgrade.</li> <li><strong>Rolling back is a restore, not a re-pin.</strong> Migrations are forward-only: pointing <code>LQMS_VERSION</code> back at the previous tag leaves the old code facing a newer schema. If an upgrade must be undone, restore the pre-upgrade backup (§9.4) <em>and</em> pin the old version. This is why the backup step is first and not optional.</li> <li><strong><code>latest</code> exists and is still the wrong answer.</strong> It always names the newest release, which means the running version depends on when the container happened to be recreated. Pin the tag: it is the only way <code>server.env</code> answers "what is running here" (ADR-0090 — a release, and an upgrade to it, is a deliberate act).</li> <li><strong>The upgrade needs the registry credential</strong> (§4.2). If <code>pull</code> returns <code>denied</code>, the token expired — the site is still up, the upgrade is simply blocked until you issue a new one.</li> </ul> <hr /> <h2 id="12-first-deploy-checklist">12. First-deploy checklist<a class="headerlink" href="#12-first-deploy-checklist" title="Permanent link">¶</a></h2> <p>The configuration was verified at <strong>config level</strong> — the merged compose output (the app resolves to the pinned <code>ghcr.io</code> image with no <code>build:</code> section and no published port, Keycloak stays loopback, a missing <code>LQMS_VERSION</code> refuses to start), the base file rendering unchanged on its own, the Caddyfile syntax, the realm substitution on both the demo and server paths, and the database role password on both paths. <strong>Nothing about the registry itself — the first publish, the package's visibility, a pull from this server — has been executed.</strong> That, and everything needing a real domain, is below. Work through it on the first deploy; nothing here is assumed to work.</p> <table> <thead> <tr> <th>#</th> <th>Check</th> <th>Command / expectation</th> </tr> </thead> <tbody> <tr> <td>1</td> <td><strong>The ghcr package is private</strong> (§4.1)</td> <td><a href="https://github.com/nburri/lqms/pkgs/container/lqms">https://github.com/nburri/lqms/pkgs/container/lqms</a> shows <strong>Private</strong>. Do this before anything else — it is the one check that cannot be undone after the fact.</td> </tr> <tr> <td>2</td> <td><strong>The server can log in to the registry</strong> (§4.2)</td> <td><code>echo '<classic PAT>' \| docker login ghcr.io -u nburri --password-stdin</code> → <code>Login Succeeded</code>, as the user that runs compose</td> </tr> <tr> <td>3</td> <td><strong>The pinned release actually pulls, and is amd64</strong></td> <td><code>docker pull ghcr.io/nburri/lqms:$LQMS_VERSION</code> then <code>docker image inspect … --format '{{.Architecture}}'</code> → <code>amd64</code>. <code>denied</code> = credential (§4.2); <code>manifest unknown</code> = the tag was never published</td> </tr> <tr> <td>4</td> <td><strong>The merged config pulls and never builds</strong></td> <td><code>./lqms-compose config</code> → the <code>app</code> service has <code>image: ghcr.io/nburri/lqms:<your tag></code>, <strong>no</strong> <code>build:</code> key, and no <code>ports:</code></td> </tr> <tr> <td>5</td> <td><strong>TLS handshake and certificate chain</strong>, both names</td> <td><code>curl -sI https://qms.example.com/api/health</code> and <code>curl -sI https://auth.qms.example.com/realms/lqms</code> → <code>200</code>, no certificate warning. Caddy's log shows <code>certificate obtained successfully</code> for both.</td> </tr> <tr> <td>6</td> <td><strong>HTTP → HTTPS redirect</strong></td> <td><code>curl -sI http://qms.example.com/</code> → <code>308</code> to <code>https://</code></td> </tr> <tr> <td>7</td> <td><strong>Health endpoint</strong></td> <td><code>curl -s https://qms.example.com/api/health</code> → <code>{"status":"UP"}</code></td> </tr> <tr> <td>8</td> <td><strong>Version stamp matches the pin</strong></td> <td><code>curl -s https://qms.example.com/api/version</code> → <code>productVersion</code> equals <code>LQMS_VERSION</code>, and the <code>sha</code> is a real short sha (not <code>unknown</code>). This is the check that proves the box runs the release you think it does.</td> </tr> <tr> <td>9</td> <td><strong>Admin console is not public</strong></td> <td><code>curl -so /dev/null -w '%{http_code}\n' https://auth.qms.example.com/admin/master/console/</code> → <code>404</code>; likewise <code>/realms/master/…</code></td> </tr> <tr> <td>10</td> <td><strong>No stray published ports</strong></td> <td><code>ss -lntp \| grep -E ':(8080\|8081\|5432)'</code> → 8081 bound to <code>127.0.0.1</code> only; nothing on 8080/5432</td> </tr> <tr> <td>11</td> <td><strong>The full OIDC round trip in a browser</strong> — the real proof</td> <td>open <code>https://qms.example.com/</code> → redirected to <code>https://auth.qms.example.com/realms/lqms/…</code> (branded login page) → sign in as your administrator → land back on the SPA, authenticated.</td> </tr> <tr> <td>12</td> <td><strong>The session survives a reload</strong></td> <td>reload the app; you stay logged in (the BFF cookie is <code>Secure</code> and was accepted — a login loop here means §13.2)</td> </tr> <tr> <td>13</td> <td><strong>Live updates work</strong> (SSE through the proxy)</td> <td>open the app in two browsers; an action in one appears in the other. This exercises <code>/api/events</code>, deliberately excluded from Caddy's compression.</td> </tr> <tr> <td>14</td> <td><strong>Logout</strong></td> <td>the Log out button ends both the app session and the Keycloak SSO session; you are asked for credentials again.</td> </tr> <tr> <td>15</td> <td><strong>Reboot survival</strong></td> <td><code>reboot</code>, then <code>docker compose -p lqms ps</code> → all services back and <code>healthy</code>.</td> </tr> <tr> <td>16</td> <td><strong>A backup, end to end</strong></td> <td>run the cron command by hand; confirm the archive exists locally <strong>and</strong> at the off-server target, and that the dead-man ping registered. Also read the manifest: <code>unzip -p backups/<archive>.zip manifest.json</code> must show a real <code>app.git_sha</code>, not <code>unknown</code> — an <code>unknown</code> there means <code>backup.sh</code> could not reach the app and every later rehearsal will fail its version check (§9.3). <strong>This check does not currently pass on the reference installation</strong> — §9.1 names both open gaps.</td> </tr> <tr> <td>17</td> <td><strong>A restore rehearsal</strong> on another machine (§9.3)</td> <td><code>verify-backup.sh</code> appends a PASS record.</td> </tr> <tr> <td>18</td> <td><strong>The registry token's expiry is in a calendar</strong> (§4.2)</td> <td>not a command — the failure mode is a blocked upgrade months from now, and nothing on the server will remind you.</td> </tr> <tr> <td>19</td> <td><strong>Invite-by-email, end to end</strong> (§5.3) — <em>only if you configured mail</em></td> <td>invite an address you control into a mandator. <strong>Two</strong> messages must arrive: LQMS's invitation with a <code>/invite/<token></code> link, then Keycloak's credential-setup link after you open it. Set the password, log in, and the invitation completes by itself. One message = one half's SMTP is missing (§5.3).</td> </tr> <tr> <td>20</td> <td><strong>The service-account client is the only one that can administer users</strong></td> <td>in the tunnelled console, <code>lqms-admin-api</code> → Service account roles shows <code>realm-management manage-users</code> and nothing else; <code>lqms-app</code> has no service account at all.</td> </tr> </tbody> </table> <p>An uptime monitor closes the loop afterwards: poll <code>https://<domain>/api/health</code> every 5 minutes and alert on anything that is not <code>200</code> with <code>"status":"UP"</code>. The endpoint is deliberately unauthenticated (ADR-0032). Add a certificate-expiry check on both names as a second signal — Caddy renews automatically, but a renewal that fails silently is the classic 60-days-later outage.</p> <hr /> <h2 id="13-troubleshooting">13. Troubleshooting<a class="headerlink" href="#13-troubleshooting" title="Permanent link">¶</a></h2> <h3 id="131-issuer-mismatch--the-classic-breakage">13.1 Issuer mismatch — the classic breakage<a class="headerlink" href="#131-issuer-mismatch--the-classic-breakage" title="Permanent link">¶</a></h3> <p><strong>Symptoms.</strong> The login page appears and accepts credentials, but the redirect back to the app ends in a <code>401</code>/<code>500</code> or an immediate bounce to the login page again. The app log shows an OIDC token verification failure naming two URLs that differ, e.g. expected <code>https://auth.qms.example.com/realms/lqms</code>, got <code>http://localhost:8081/realms/lqms</code>. A bearer-token call (<code>/api/me</code>) fails the same way with a token that Keycloak just issued.</p> <p><strong>Cause.</strong> Three values must name the <em>same</em> issuer:</p> <table> <thead> <tr> <th>Value</th> <th>Set by</th> <th>Correct value</th> </tr> </thead> <tbody> <tr> <td>what Keycloak stamps as <code>iss</code></td> <td><code>KC_HOSTNAME</code></td> <td><code>https://auth.<domain></code></td> </tr> <tr> <td>what the app expects</td> <td><code>QUARKUS_OIDC_TOKEN_ISSUER</code></td> <td><code>https://auth.<domain>/realms/lqms</code></td> </tr> <tr> <td>where the app fetches discovery/JWKS</td> <td><code>QUARKUS_OIDC_AUTH_SERVER_URL</code></td> <td><code>http://keycloak:8080/realms/lqms</code> — <strong>stays internal, on purpose</strong></td> </tr> </tbody> </table> <p>The third one looks wrong and is not: <code>KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true</code> makes Keycloak serve a discovery document whose <em>browser-facing</em> endpoints (<code>issuer</code>, <code>authorization_endpoint</code>, <code>end_session_endpoint</code>) carry the public HTTPS name while <code>token_endpoint</code> and <code>jwks_uri</code> follow the requesting host. Verified against 26.4 with exactly this configuration.</p> <p><strong>Fix.</strong> Do not "repair" it by pointing <code>auth-server-url</code> at the public URL. Check instead that <code>LQMS_DOMAIN</code> is identical everywhere (<code>./lqms-compose config | grep -E 'KC_HOSTNAME|TOKEN_ISSUER'</code>), that both DNS names resolve to this host, and that Keycloak has <code>KC_PROXY_HEADERS=xforwarded</code> so it sees the original <code>https</code> scheme. Then:</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-28-1" name="__codelineno-28-1" href="#__codelineno-28-1"></a><span class="c1"># the app image ships no curl (its healthcheck uses bash /dev/tcp), so ask from a throwaway</span> <a id="__codelineno-28-2" name="__codelineno-28-2" href="#__codelineno-28-2"></a><span class="c1"># container ON THE COMPOSE NETWORK — i.e. from exactly where the app asks</span> <a id="__codelineno-28-3" name="__codelineno-28-3" href="#__codelineno-28-3"></a>docker<span class="w"> </span>run<span class="w"> </span>--rm<span class="w"> </span>--network<span class="w"> </span>lqms_default<span class="w"> </span>curlimages/curl:latest<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span> <a id="__codelineno-28-4" name="__codelineno-28-4" href="#__codelineno-28-4"></a><span class="w"> </span>http://keycloak:8080/realms/lqms/.well-known/openid-configuration<span class="w"> </span><span class="se">\</span> <a id="__codelineno-28-5" name="__codelineno-28-5" href="#__codelineno-28-5"></a><span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span><span class="s1">','</span><span class="w"> </span><span class="s1">'\n'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-E<span class="w"> </span><span class="s1">'"issuer"|authorization_endpoint'</span> </code></pre></div> <p><code>issuer</code> must be <code>https://auth.<domain>/realms/lqms</code>, character for character.</p> <p><strong>If you changed the domain after the realm was imported</strong>, Keycloak keeps the stored realm; the import does not re-run. Update <code>KC_HOSTNAME</code> via <code>server.env</code> <strong>and</strong> fix the client's redirect URIs and web origins in the admin console (or with <code>kcadm</code>) — otherwise Keycloak rejects the redirect with <code>Invalid parameter: redirect_uri</code>. Identities themselves survive a domain change: they bind on the Keycloak subject, not the issuer (ADR-0063).</p> <h3 id="132-login-loop--the-session-cookie-is-dropped">13.2 Login loop — the session cookie is dropped<a class="headerlink" href="#132-login-loop--the-session-cookie-is-dropped" title="Permanent link">¶</a></h3> <p><strong>Symptoms.</strong> Keycloak authenticates you, the browser returns to the app, and the app immediately redirects to the login page again, forever. No error is displayed.</p> <p><strong>Cause.</strong> The BFF session cookie is <code>Secure</code> and something in the chain is not HTTPS: the app built an <code>http://</code> redirect URI (missing <code>X-Forwarded-Proto</code>), or someone set <code>LQMS_COOKIE_SECURE=false</code> from the LAN-demo instructions.</p> <p><strong>Fix.</strong> Confirm the app sees the proxy headers — <code>QUARKUS_HTTP_PROXY_PROXY_ADDRESS_FORWARDING</code>, <code>QUARKUS_HTTP_PROXY_ALLOW_X_FORWARDED</code> and <code>QUARKUS_HTTP_PROXY_ENABLE_FORWARDED_HOST</code> are all <code>true</code> in the merged config — and that <code>QUARKUS_OIDC_AUTHENTICATION_COOKIE_FORCE_SECURE</code> is <code>true</code>. If a redirect URI still comes out as <code>http://</code>, force it: <code>QUARKUS_OIDC_AUTHENTICATION_FORCE_REDIRECT_HTTPS_SCHEME=true</code>.</p> <h3 id="133-certificates-are-not-issued">13.3 Certificates are not issued<a class="headerlink" href="#133-certificates-are-not-issued" title="Permanent link">¶</a></h3> <p>Caddy logs <code>could not get certificate from issuer</code> or repeated ACME failures.</p> <ul> <li>Both names must resolve to <strong>this</strong> host: <code>dig +short <name></code>.</li> <li>Port 80 must reach Caddy from the internet (the HTTP-01 challenge). A cloud security group is a second firewall — check it as well as ufw.</li> <li>An AAAA record without working IPv6 breaks validation; remove it or fix routing.</li> <li>Let's Encrypt rate-limits failures (5 per account per hour). While debugging, add <code>acme_ca https://acme-staging-v02.api.letsencrypt.org/directory</code> to the Caddyfile's global block — staging certificates are untrusted (expect a browser warning) but unlimited. Remove it afterwards and delete <code>/data/caddy/acme</code> inside the caddy volume so production certificates are requested.</li> <li>Certificates live in the <code>lqms-caddy-data</code> volume. Deleting that volume forces re-issuance and can run you into the rate limit — it is part of the deployment, not a cache.</li> </ul> <h3 id="134-exec-format-error-when-the-app-container-starts">13.4 <code>exec format error</code> when the app container starts<a class="headerlink" href="#134-exec-format-error-when-the-app-container-starts" title="Permanent link">¶</a></h3> <p>The image was built for another architecture. On the supported path this cannot happen — CI publishes <code>linux/amd64</code> only (§4) — so seeing it means the running image is <strong>not</strong> the published one: somebody built it by hand on an Apple Silicon Mac (§4.4) and <code>docker load</code>ed it under the same tag. Confirm with</p> <div class="highlight"><pre><span></span><code><a id="__codelineno-29-1" name="__codelineno-29-1" href="#__codelineno-29-1"></a>docker<span class="w"> </span>image<span class="w"> </span>inspect<span class="w"> </span><span class="s2">"ghcr.io/nburri/lqms:</span><span class="nv">$LQMS_VERSION</span><span class="s2">"</span><span class="w"> </span>--format<span class="w"> </span><span class="s1">'{{.Architecture}}'</span><span class="w"> </span><span class="c1"># -> amd64</span> </code></pre></div> <p>which must match <code>uname -m</code> on the server (<code>amd64</code> ↔ <code>x86_64</code>). The fix is to delete the local image and pull the published one: <code>docker image rm</code> then <code>./lqms-compose pull app</code>.</p> <h3 id="135-the-app-refuses-to-boot-fan-out-deployment-check">13.5 The app refuses to boot: fan-out deployment check<a class="headerlink" href="#135-the-app-refuses-to-boot-fan-out-deployment-check" title="Permanent link">¶</a></h3> <p><code>FanoutDeploymentCheck</code> fails when the migration role cannot bypass RLS (ADR-0035 §7). On a server this almost always means the PostgreSQL volume was initialised <strong>without</strong> <code>postgres-init.sql</code> — for example by a <code>docker compose up</code> that used only the base file in a directory where the mount path resolved differently. <code>deployment.md</code> §4 has the recovery. The init script runs <strong>only</strong> on an empty data volume.</p> <h3 id="136-database-authentication-failure-after-a-restore">13.6 Database authentication failure after a restore<a class="headerlink" href="#136-database-authentication-failure-after-a-restore" title="Permanent link">¶</a></h3> <p><code>FATAL: password authentication failed for user "lqms_migration"</code>. The restore recreated the PostgreSQL volume with the demo password while the app uses the generated one — §9.4 has the <code>ALTER ROLE</code> step.</p> <h3 id="137-http-431--a-wedged-browser-profile">13.7 HTTP 431 / a wedged browser profile<a class="headerlink" href="#137-http-431--a-wedged-browser-profile" title="Permanent link">¶</a></h3> <p>Documented in <code>deployment.md</code> §5.5 (finding #18): clear the site's cookies. It fails clean by design; never treat it by raising the header limit.</p> <h3 id="138-the-image-will-not-pull">13.8 The image will not pull<a class="headerlink" href="#138-the-image-will-not-pull" title="Permanent link">¶</a></h3> <table> <thead> <tr> <th>What the pull says</th> <th>What it means</th> </tr> </thead> <tbody> <tr> <td><code>denied</code> / <code>unauthorized</code></td> <td>no usable credential for a <strong>private</strong> package. Either no <code>docker login</code> on this box, or a login as a different user than the one running compose (root vs. you — §4.2), or the token expired, or the token is a <em>fine-grained</em> one, which <code>ghcr.io</code> does not accept. Re-issue a <strong>classic</strong> token with <code>read:packages</code>.</td> </tr> <tr> <td><code>manifest unknown</code></td> <td><code>LQMS_VERSION</code> names a tag that was never published. Check the package's Versions list; remember a <code>workflow_dispatch</code> without <code>publish_tag</code> publishes <code><ref>-<sha></code>, not a version tag.</td> </tr> <tr> <td><code>no matching manifest for linux/amd64</code></td> <td>you are pulling on a non-amd64 host. Only amd64 is published (§4).</td> </tr> <tr> <td>compose starts a <strong>build</strong> instead of pulling</td> <td>you ran a base-file-only command (<code>restore.sh</code>, <code>verify-backup.sh</code>, a bare <code>docker compose up</code>) — §7.1. Export <code>LQMS_IMAGE</code>, or use the <code>lqms-compose</code> wrapper, which always loads both files.</td> </tr> </tbody> </table> <p>The stack keeps running through all of these: an image that cannot be pulled blocks an upgrade, it does not stop containers that are already up.</p> <hr /> <h2 id="14-what-this-deployment-deliberately-does-not-do">14. What this deployment deliberately does not do<a class="headerlink" href="#14-what-this-deployment-deliberately-does-not-do" title="Permanent link">¶</a></h2> <p>Named so that none of it reads as an oversight:</p> <ul> <li><strong>Keycloak still uses the <code>dev-file</code> (H2) database.</strong> It runs in production mode (<code>start</code>, not <code>start-dev</code>) with a strict hostname, but its store is a single file on the <code>lqms-kc</code> volume. It is backed up as a realm export including users on every run of <code>backup.sh</code> (ADR-0060), and identities bind to Keycloak subjects (ADR-0063) — so that volume is load-bearing. Moving Keycloak onto PostgreSQL is a separate piece of work.</li> <li><strong>The app still runs on the single BYPASSRLS datasource.</strong> Row-level security is bypassed at runtime and tenant separation rests on the application layer, exactly as in the demo stack. The two-role hardening is specified in <code>deployment.md</code> §4 and is not applied by this override.</li> <li><strong>No SMTP.</strong> Notification email is disabled (ADR-0041); there is no mail channel to configure yet.</li> <li><strong>No log aggregation, no metrics stack.</strong> Container logs are size-capped (10 MB × 3 per service) and read with <code>docker compose logs</code>. The health endpoint plus the backup dead-man ping are the whole monitoring story — and on the reference installation the ping half is not configured yet (§9.1, gap 2), so today the health endpoint is the only live signal there.</li> <li><strong>Caddy access logs are off.</strong> Enabling them records client IPs and document URLs — personal data needing a retention decision. The Caddyfile shows how, deliberately commented out.</li> <li><strong><code>smoke.sh</code> was not adapted to run against a public URL.</strong> It builds and tears down with <code>down -v</code>; making it server-safe is more than a parameter. §12 is the checklist that replaces it.</li> <li><strong>The published image is neither signed nor attested.</strong> No cosign signature, and the workflow deliberately turns provenance/SBOM attestations off (the reason is in its comments). What the image <em>does</em> carry is the ADR-0090 identity — <code>productVersion</code> + git sha at <code>/api/version</code> — and the release SBOM the <code>sbom</code> job attaches to the GitHub release. The image's own layer (L3, <code>syft</code>) is the registered SBOM follow-up now that a job builds the image at all.</li> <li><strong>The registry credential is one personal classic token.</strong> GHCR offers nothing narrower than an account-wide <code>read:packages</code> scope (§4.2), so the deployment's pull right is coupled to one person's account. A dedicated machine account is the hardening when this stops being a single-owner setup.</li> </ul> </article> </div> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type="button" class="md-top md-icon" data-md-component="top" hidden> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class="md-footer"> <nav class="md-footer__inner md-grid" aria-label="Footer" > <a href="../deployment/" class="md-footer__link md-footer__link--prev" aria-label="Previous: LQMS Deployment Guide"> <div class="md-footer__button md-icon"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg> </div> <div class="md-footer__title"> <span class="md-footer__direction"> Previous </span> <div class="md-ellipsis"> LQMS Deployment Guide </div> </div> </a> <a href="../../traceability/" class="md-footer__link md-footer__link--next" aria-label="Next: Requirements Traceability"> <div class="md-footer__title"> <span class="md-footer__direction"> Next </span> <div class="md-ellipsis"> Requirements Traceability </div> </div> <div class="md-footer__button md-icon"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11z"/></svg> </div> </a> </nav> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class="md-copyright"> <div class="md-copyright__highlight"> Built from <code>v0.9.56-4-g06c7f811</code> (commit <code>06c7f811</code>, 2026-09-09) — the same Markdown the repository ships. </div> </div> </div> </div> </footer> </div> <div class="md-dialog" data-md-component="dialog"> <div class="md-dialog__inner md-typeset"></div> </div> <script id="__config" type="application/json">{"base": "../..", "features": ["navigation.instant", "navigation.tracking", "navigation.top", "navigation.footer", "navigation.indexes", "toc.follow", "search.suggest", "search.highlight", "content.code.copy", "content.action.edit"], "search": "../../assets/javascripts/workers/search.6ce7567c.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}}</script> <script src="../../assets/javascripts/bundle.83f73b43.min.js"></script> </body> </html>