7. Deployment View¶
The full operator-facing guide is docs/operations/deployment.md (every configuration key, IdP integration, the bootstrap flow, sweep tuning, upgrades); this section states the architecturally significant shape.
Deployment form (as built)¶
- Backend: a single OCI container image built by the multi-stage
deploy/Dockerfile. The build stage (eclipse-temurin:21-jdk-noble - Node) runs
./gradlew :backend:api:quarkusBuild -Dquarkus.quinoa=true; the runtime stage is a Quarkus JVM fast-jar oneclipse-temurin:25-jre-noble, run as a non-rootlqmsuser, listening on0.0.0.0:8080. A GraalVM native image is not built — the JVM fast-jar is the shipped form (TBD if native is ever pursued). jOOQ codegen needs a Docker socket the hermetic image build lacks, so pre-generated jOOQ sources are fed in via-Plqms.jooq.provided(deployment.md §2 — the operator regenerates them when migrations change). - Frontend: the Angular SPA is bundled same-origin into the backend jar by Quarkus Quinoa
(ADR-0032) —
/triggers OIDC login,/api/*is the authenticated API, static assets are public; no separate frontend deployment. - Database: PostgreSQL (the reference stack pins 18), holding all state under FORCE row-level security.
- Document files: a server-filesystem content store at
LQMS_STORAGE_ROOT(a mounted volume;/app/data/contentby default), S3-compatible store as a future option (ADR-0001). - Authentication: an external OIDC provider — a corporate IdP or the bundled Keycloak (the reference stack pins 26.4); LQMS stores no credentials (ADR-0011).
- LLM (search layers 2–3): cloud-based or on-prem per mandator policy — future work (§8.7 / §8 security concepts).
- Reference stack:
deploy/docker-compose.ymlwires app + PostgreSQL + Keycloak with named volumes (lqms-db,lqms-kc,lqms-content); it is a demo/colleague-feedback topology (demo secrets, single datasource), not the hardened production posture below.
browser ──TLS──> [ reverse proxy / TLS term ] ──> LQMS app (:8080, JVM fast-jar on Temurin 25-jre)
│ ├── PostgreSQL 18 (data + FORCE RLS)
│ └── content store (filesystem volume)
──OIDC──> IdP (corporate or Keycloak) <─────┘ (discovery, JWKS, token, login redirect)
Database roles — the BYPASSRLS requirement and the split-role hardening¶
Mandator/project separation is enforced in the database: every content table is FORCE RLS, and
the cross-tenant fan-out (lqms_fanout_source_event, SECURITY DEFINER) can only cross scope
boundaries if its owner — the role that ran the migrations — is superuser or has BYPASSRLS.
A plain owner makes the fan-out fail closed for separation but silent for compliance (derived
tenants are never notified). The app therefore runs FanoutDeploymentCheck at startup and
refuses to boot if the owner cannot bypass RLS (ADR-0035 §7).
The reference stack is single-datasource: the app connects with the same BYPASSRLS migration
role it migrates with, but runtime RLS is still enforced for content access — every content
transaction enters the NOBYPASSRLS role via SET LOCAL ROLE lqms_app (RlsScopeContext, ADR-0026)
before touching data. The hardened production posture splits this into a BYPASSRLS
lqms_migration role (Flyway + DEFINER owner) and a separate NOBYPASSRLS lqms_runtime datasource,
so a query that forgets its scope filter fails closed at the database. That split is staged, not yet
implemented — see deployment.md §4 "Two-role hardening"
and the accepted defense-in-depth debt in §11.
Backup zips must be restorable on the same or a different server (see §7.1).
7.1 Backup & restore¶
Realizes ADR-0060 (mechanism B-1a, restore
rehearsal B-2a, disposal interplay B-3) against the reference stack in deploy/docker-compose.yml.
Regulatory anchor: ISO 13485 §4.2.4 g) (prevent loss of documents) and §4.2.5 (records stay
retrievable). For a QMS that lives in LQMS the backup system is that control, so restore is
proven weekly, not assumed.
Three deploy-side scripts, all Bash 3.2 compatible (they run on the operator's machine; the heavy lifting happens inside containers):
| Script | Role |
|---|---|
deploy/backup.sh |
Nightly full backup → one self-contained zip (REQ-BAK-001). |
deploy/verify-backup.sh |
Weekly restore rehearsal into a throwaway scratch project; writes a dated verification record. |
deploy/restore.sh |
Operator restore of an archive into a compose project, with the post-restore obligations. |
What a full backup contains and why the order matters¶
An archive holds, in this order: content blobs first, then the database dump, then the Keycloak
realm export, then a manifest (REQ-BAK-001 inventory). The ordering is the consistency mechanism
(ADR-0001): blobs are immutable and SHA-256-named, so blobs-first, database-second makes the
archive self-consistent without stopping the application — a dangling DB→blob reference can only
arise in the reverse order. Every copied blob is re-hashed against its filename during the copy
(REQ-BAK-004); any mismatch fails the backup loudly and doubles as bit-rot detection on the primary
store. Keycloak's dev-mode H2 file locks while start-dev runs, so the realm export (users included,
via kc.sh export) is the one step that briefly pauses Keycloak — the app and its data keep running
(ADR-0060 D-4; the production-shaped answer is Keycloak on PostgreSQL, which dissolves this special
case). No secrets go into the archive (ADR-0060 §1 item 5): DB passwords, the OIDC client secret
and the Keycloak admin password are re-provisioned on restore.
The manifest records: creation timestamp, the app git SHA + build timestamp (from /api/version),
blob count / total bytes / a hash sample, the dump and realm-export SHA-256s, the retention window,
and tool versions.
Schedule examples¶
Nightly backup (RPO ≤ 24 h, REQ-NFR-004) and weekly rehearsal. Set LQMS_BACKUP_TARGET for the
off-box copy (ADR-0018 — "stored separately from the primary system") and LQMS_BACKUP_PING_URL
for the dead-man success ping (absence of the ping is the alert).
cron (Linux host):
# nightly full backup at 02:15 — a USER crontab (lqms-ops), matching the reference installation
# (server-deployment.md §9.1). The off-box target and dead-man ping shown here are the INTENDED
# shape; on the reference installation both are still pending host-side setup (§9.1 names them).
15 2 * * * cd /home/lqms-ops/lqms/deploy && LQMS_BACKUP_TARGET=/mnt/nas/lqms LQMS_BACKUP_PING_URL=https://hc-ping.com/<uuid> ./backup.sh >> /home/lqms-ops/lqms-backup.log 2>&1
# weekly restore rehearsal, Sundays 03:30 (scratch project, distinct ports) — on a SEPARATE machine
# where possible (§9.3)
30 3 * * 0 cd /home/lqms-ops/lqms/deploy && ./verify-backup.sh >> /home/lqms-ops/lqms-verify.log 2>&1
launchd (macOS) — one StartCalendarInterval agent per script; the backup agent's ProgramArguments
is equivalent to the cron line:
<key>ProgramArguments</key>
<array>
<string>/bin/bash</string>
<string>-lc</string>
<string>cd /Users/you/lqms/deploy && LQMS_BACKUP_TARGET=/Volumes/backup/lqms ./backup.sh</string>
</array>
<key>StartCalendarInterval</key>
<dict><key>Hour</key><integer>2</integer><key>Minute</key><integer>15</integer></dict>
Retention (B-3 — disposal ↔ backup interplay)¶
Local archives are pruned past a bounded retention window, default 35 days
(LQMS_BACKUP_RETENTION_DAYS, REQ-DPR-011 / ADR-0025). The customer-facing statement:
Disposed content persists in backups for at most the retention window (35 days by default, configurable). After disposal, content lingers in existing archives only until they age out of that window. Legal holds do NOT extend backups (ADR-0040): a hold prevents disposal upstream in the live system; it does not preserve or lengthen the retention of archives.
Pruning the off-box copies to the same window is the operator's responsibility on the destination
(the script prunes the local copy; see LQMS_BACKUP_TARGET).
Restore runbook¶
- Obtain the archive (from the off-box target if the primary host is lost) onto a host with the compose stack and a same-or-newer app image (Flyway migrates forward only — never backward).
cd deploy && ./restore.sh <archive.zip> [--project NAME] [--force]. It refuses to run against a project that already has volumes unless--force(which wipes them). It then, in order: loads the blobs into a fresh content volume, starts PostgreSQL on a fresh volume (postgres-init.sqlrecreates the roles),pg_restores the dump (ownership + grants preserved, so objects stay owned by the BYPASSRLSlqms_migrationrole per ADR-0035 §7), imports the Keycloak realm with users into a fresh Keycloak volume, and starts the app (which Flyway-migrates forward).- Mandatory RE-DISPOSAL step (regulatory — ADR-0060 §Decision 3, ADR-0025). A restore of a
pre-disposal archive resurrects content that was disposed after the archive's timestamp. The
disposal audit trail survives disposal, so it is the source of truth: compare the restored DB's
disposal log against the latest known disposal log and re-execute every disposal recorded after
the archive's
created_at.restore.shprints this obligation on every run; it is not optional. - Re-provision secrets (they are never in the archive): PostgreSQL superuser +
lqms_migrationpassword, Keycloak admin password, thelqms-app-secretOIDC client secret (must match in compose and the realm), and every real user password. Seedeploy/README.mdand this document's demo caveats. - Verify end-to-end (
deploy/smoke.sh) and reindex search if it was excluded from the dump (REQ-BAK-003 permits exclude+rebuild; the current dump includes it). Target: within RTO ≤ 1 business day (ADR-0018) — the weekly rehearsal continuously proves the mechanical part takes minutes, not days.
Restore rehearsal (the differentiator, B-2a)¶
verify-backup.sh restores the newest archive into a scratch compose project (distinct project
name, distinct published ports via LQMS_VERIFY_APP_PORT / LQMS_VERIFY_KC_PORT, distinct volumes)
that runs alongside the primary stack, then verifies: the app is healthy and /api/version matches
the manifest SHA; blob re-hash is clean over the restored store; and referential closure — every
content_part blob reference in the restored DB resolves to a file (the corruption direction the
ordering rule prevents, verified rather than trusted). The scratch project is always torn down, and a
dated record line is appended to deploy/backup-verification.log (gitignored) and printed. That
record chain is the audit answer to "when did you last prove a restore?" and is the recurring
objective evidence the CSV validation package cites (ADR-0061).
Keycloak-volume continuity (ADR-0063 dependency)¶
Identities bind on the Keycloak subject (user UUID) alone, not on the issuer (ADR-0063), so a
change of host/scheme/proxy keeps users' roles and documents. The one hard continuity requirement is
that Keycloak's state survives — subjects are Keycloak user UUIDs, so identities live only as
long as Keycloak's data does. That is exactly why the realm export (users included) rides every
backup and is re-imported on every restore: losing the Keycloak state orphans every user under
any key design. When Keycloak moves to a PostgreSQL backing store (ADR-0060 D-4), its state rides the
same pg_dump and this dedicated export step disappears.