# State, events and recovery Nonfinal architecture review · 0.4.0. Generated from `assets/fixtures-architecture.js`; edit that source and rerun `node tools/generate-docs.js`. A database sequence is not a committed cursor. This design serializes writers through a transactional row. No finite overlap is used. ## definitions ### writer Lock the cursor row for tenant_id + catalog_id + history_epoch BEFORE changing exported state. Update the transactional counter, accepted rows and outbox in one transaction. Rollback consumes no position. A delayed writer blocks later commits in this stream. ### retention Prune only a contiguous prefix of published events whose retention age has elapsed, through the minimum of every registered live consumer’s durable contiguous progress and every unexpired rebuild lease pin. Serialize lease creation/advancement, consumer registration and pruning with the retention-control lock. Delivery or acknowledgement alone is not application progress; a missing position or parked dependency protects the tail. Monitor blocked retention and lease expiry; never silently remove a lagging live consumer. ### live Progress Commit projection changes, inbox outcomes, dependency parking and the greatest contiguous processed position together in the live product transaction. Persist/report progress monotonically to resolver retention control for the same tenant, catalog and history epoch only after that commit; an interrupted report conservatively retains extra events. The pruner may use only acknowledged durable progress. Shadow catch-up has a separate cursor and lease; it never advances the live cursor. Cutover atomically swaps serving generation and durable live cursor before releasing the rebuild lease. ### identity Lookup Join on (tenant_id, catalog_id, class_id, identity_generation, identity_hash), using the capture’s pinned active generation; replay uses its originally pinned generation. Candidate/backfill rows are separate and cannot serve active lookup before activation. Strong identifiers narrow candidates but do not bypass generation-qualified identity. ### identity Uniqueness UNIQUE (tenant_id, catalog_id, class_id, identity_generation, identity_hash) for ready/active keys. Candidate collision rows remain separate. Different generations may coexist; activation requires completed backfill, resolved collisions/splits and human approval. ### queue Continuity Accepted-change committed positions are contiguous within (tenant_id, catalog_id, history_epoch). Writers allocate them with state and outbox under the transactional cursor lock; rollback consumes no position. Queue arrivals may be unordered, duplicated or temporarily missing, so consumers park dependencies and retrieve retained events to close gaps. Durable progress advances only through every processed position, including safely ignored stale revisions. A DLQ entry or an ACK of a parked event does not close a gap. Internal resolution/replay jobs use their own dedupe IDs and do not share this committed cursor. ### receipt Ownership Collectors read receipts owned by their stable instance_id, not the rotating OAuth client_id. Credential rotation preserves legitimate receipt access. Authenticated unknown receipt: 404 NOT_FOUND; another tenant: 403 TENANT_SCOPE_DENIED; same tenant/source but another instance: 403 INSTANCE_SCOPE_DENIED. Operators use their tenant-scoped viewer grant. ## invariants ### id INV-01 ### status PROP ### text A listing mapping revision targets a catalog item that exists and is active at that mapping revision. ### enforce Same transaction: lock target item row (FOR SHARE), check status; event carries dependency [catalog_item, id, rev]. ### id INV-02 ### status PROP ### text One active item per normalized identity key within tenant, catalog, class and identity-rule generation. ### enforce UNIQUE (tenant_id, catalog_id, class_id, identity_generation, identity_hash) for ready/active keys. Candidate collision rows remain separate. Different generations may coexist; activation requires completed backfill, resolved collisions/splits and human approval. ### id INV-03 ### status PROP ### text Shipping policy versions have non-overlapping effective windows per scope. ### enforce Exclusion constraint on (tenant_id, policy_id, scope) with tstzrange(effective_from, effective_to) using && (needs btree_gist; verify on the managed provider). ### id INV-04 ### status EST ### text A decision is made against a specific review-case revision; a stale revision is rejected. ### enforce UPDATE review_case ... WHERE revision = expected; zero rows → 409 (prototype behaviour). ### id INV-05 ### status EST ### text Every accepted fact references a decision and the evidence (capture + locator) it rests on. ### enforce NOT NULL foreign keys to decision; evidence refs array non-empty (check constraint). ### id INV-06 ### status EST ### text Every accepted change produces outbox rows in the same transaction. ### enforce UnitOfWork collects domain events and writes outbox rows before COMMIT; repositories cannot commit on their own. ### id INV-07 ### status PROP ### text An attribute value references a published definition version whose applicability includes the subject class (directly or inherited). ### enforce Foreign key to (definition key, version); applicability checked in the domain service and by a deferred constraint trigger (choice OPEN). ### id INV-08 ### status PROP ### text A split never deletes the source item and never rewrites prior decisions; successors and remaps reference them. ### enforce No DELETE grant on catalog_item or decision for resolver_app; split recorded as status + successor rows. ## atomic Writes ### id AW-01 ### title Resolver decision (review or automatic rule) ### owner mod_review / mod_identity_relationships ### status PROP ### stages - BEGIN - resolver tx (READ COMMITTED + row locks; isolation choice validated in SPK-01) - PROP - Coordinate stream writer - Lock the cursor row for tenant_id + catalog_id + history_epoch BEFORE changing exported state. Update the transactional counter, accepted rows and outbox in one transaction. Rollback consumes no position. A delayed writer blocks later commits in this stream. - PROP - Check case revision - UPDATE review_case SET revision = revision + 1 WHERE id AND revision = expected (INV-04) - EST - Insert decision - Immutable row: action, actor, rationale, evidence refs, prior decision refs - PROP - Apply facts - Item / mapping / attribute rows with revision + 1; check INV-01/02/07 - PROP - Insert outbox rows - Allocate consecutive committed positions with the transactional cursor; one event per affected aggregate revision. Include tenant, catalog, history_epoch, event id, revision, dependencies and payload in the same transaction as decision and facts. - EST - COMMIT - All or nothing - EST - Relay (after commit) - Publishes outbox rows to the product-ingestion queue; duplicates possible - EST ### id AW-02 ### title Capture intake ### owner mod_intake_evidence ### status PROP ### stages - Authenticate + validate - Capture schema version, idempotency key (credential model: see Trust & tenancy) - PROP - Put artifact - Content-addressed, put-if-absent in the evidence bucket (before the tx) - PROP - BEGIN - resolver tx - PROP - Insert capture + receipt + job row - Duplicate idempotency key → return the existing receipt - PROP - COMMIT - Receipt is now durable - PROP - Enqueue job (ref) - After commit; on failure the pending-job sweeper re-enqueues - PROP ### id AW-03 ### title Outbox relay ### owner svc_resolver (relay) ### status PROP ### stages - Read batch - SELECT unpublished rows ORDER BY seq LIMIT batch (FOR UPDATE SKIP LOCKED if several relays) - PROP - Publish - Send batch to the product-ingestion queue; message carries event id + small payload or ref - PROP - Mark published - UPDATE published_at; crash between publish and mark → re-publish (duplicate) - PROP - Prune - Prune only a contiguous prefix of published events whose retention age has elapsed, through the minimum of every registered live consumer’s durable contiguous progress and every unexpired rebuild lease pin. Serialize lease creation/advancement, consumer registration and pruning with the retention-control lock. Delivery or acknowledgement alone is not application progress; a missing position or parked dependency protects the tail. Monitor blocked retention and lease expiry; never silently remove a lagging live consumer. - PROP ### id AW-04 ### title Product consumer apply ### owner svc_product ### status EST ### stages - Receive - At-least-once delivery (Queues, verified 2026-10-12) - EST - BEGIN - product tx - EST - Inbox insert - INSERT inbox(event_id) ON CONFLICT DO NOTHING; conflict → duplicate → COMMIT + ack - EST - Revision check - Event rev ≤ applied rev for the aggregate → record as ignored - EST - Dependency check - Missing dependency revision → park the event (parked row) and ack; released when the dependency applies - EST - Apply projection - Upsert projection rows; set applied rev - EST - Commit contiguous progress - Commit projection changes, inbox outcomes, dependency parking and the greatest contiguous processed position together in the live product transaction. Persist/report progress monotonically to resolver retention control for the same tenant, catalog and history epoch only after that commit; an interrupted report conservatively retains extra events. The pruner may use only acknowledged durable progress. Shadow catch-up has a separate cursor and lease; it never advances the live cursor. Cutover atomically swaps serving generation and durable live cursor before releasing the rebuild lease. - PROP - COMMIT, then ack - Ack only after commit; crash before ack → redelivery → inbox dedupe - EST ## bootstrap - Serialize writers - Lock the cursor row for tenant_id + catalog_id + history_epoch BEFORE changing exported state. Update the transactional counter, accepted rows and outbox in one transaction. Rollback consumes no position. A delayed writer blocks later commits in this stream. - PROP - Pin retention - Under the retention-control lock shared by the pruner, create a renewable rebuild lease pinned after current pruned-through cursor P. Commit before opening the snapshot. - PROP - Read one snapshot - REPEATABLE READ READ ONLY: read history_epoch, committed stream cursor W, and all accepted state including private/public releases and aggregate revisions in one snapshot. Never use max(outbox.sequence) or a PostgreSQL sequence. - PROP - Publish export - Advance lease pin from P to W under retention coordination. Write checksummed export objects. Manifest becomes ready only after export completion and a valid lease check; abort on epoch change. - PROP - Load shadow - Load the manifest into an isolated projection generation, with aggregate revisions and W as its starting contiguous applied position. Renew the lease throughout export/load/catch-up. - PROP - Retrieve retained events - GET /v1/exports/events?after=W&lease_id=…&history_epoch=… returns ordered authorized stream events. The pruner cannot remove events after an active pin. A stale cursor or expired lease returns 410; restart from a new export. - PROP - Catch up and cut over - Choose committed target H. Apply every position through H, retaining dedupe, revisions and dependency parking. Require no unresolved dependencies at/below H. Pause the local applier, atomically swap the serving generation and durable consumer cursor, resume after H, then release the lease. - PROP - Advance live retention progress - Commit projection changes, inbox outcomes, dependency parking and the greatest contiguous processed position together in the live product transaction. Persist/report progress monotonically to resolver retention control for the same tenant, catalog and history epoch only after that commit; an interrupted report conservatively retains extra events. The pruner may use only acknowledged durable progress. Shadow catch-up has a separate cursor and lease; it never advances the live cursor. Cutover atomically swaps serving generation and durable live cursor before releasing the rebuild lease. - PROP - Fence restored history - Pause writers, relay, consumers and public serving; issue a NEW epoch in a trusted control plane outside the restored database. Fence old workers; abandon old events without relabeling; initialize a new cursor and rebuild from restored state. Audit and terminally ignore old-epoch deliveries. - PROP ## retention ### data Evidence artifacts (HTML, JSON, screenshots) ### store Private evidence bucket ### retention Kept while referenced by accepted facts, decisions, open cases or regression fixtures; unreferenced expire after a window (DEC-06) ### status OPEN ### recovery Object-store durability + versioning/replication choice OPEN; not rebuildable ### rpo OPEN ### rto OPEN ### data Claims, extraction runs, unmapped fields ### store resolver schema ### retention As long as the artifact they cite (PROP) ### status PROP ### recovery Database PITR; also re-derivable by replay from retained artifacts ### rpo Same as resolver DB ### rto Same as resolver DB ### data Accepted state (items, mappings, policies) ### store resolver schema ### retention Permanent (current + revisions) ### status PROP ### recovery Database PITR ### rpo OPEN (DJ-04) ### rto OPEN (DJ-04) ### data Decision history + review cases ### store resolver schema ### retention Permanent; immutable (DJ-08) ### status PROP ### recovery Database PITR ### rpo OPEN (DJ-04) ### rto OPEN (DJ-04) ### data Integration events (outbox) ### store resolver.outbox ### retention Prune only a contiguous prefix of published events whose retention age has elapsed, through the minimum of every registered live consumer’s durable contiguous progress and every unexpired rebuild lease pin. Serialize lease creation/advancement, consumer registration and pruning with the retention-control lock. Delivery or acknowledgement alone is not application progress; a missing position or parked dependency protects the tail. Monitor blocked retention and lease expiry; never silently remove a lagging live consumer. ### status PROP ### recovery Not backed up separately; lost tail is recovered by snapshot rebuild ### rpo n/a ### rto n/a ### data Product inbox + parked events ### store product schema ### retention At least the outbox window so late duplicates are still caught ### status PROP ### recovery Rebuilt with the projection ### rpo n/a ### rto n/a ### data Product projections + search docs ### store product schema / index ### retention Current only ### status PROP ### recovery Rebuild from snapshot + tail (ADR-004); backups optional ### rpo n/a (derived) ### rto Rebuild time — to measure (SPK-05) ### data Snapshot exports ### store Object storage ### retention Keep the last few successful exports (count OPEN) ### status OPEN ### recovery Re-export ### rpo n/a ### rto n/a ### data Database backups ### store Managed provider PITR ### retention Provider window, chosen with DJ-04 ### status OPEN ### recovery Restore while stopped; rotate externally trusted history epoch; rebuild projections; discard old-epoch deliveries ### rpo OPEN ### rto OPEN ## recovery - PROP|Every projection-affecting write uses the cursor lock; unrelated evidence intake need not. Per-stream serialization is a throughput tradeoff to measure. - PROP|Pruning respects active rebuild pins AND the durable live-consumer cursor. Expired leases cannot be revived; a rebuild restarts. Lease duration and retained-byte budgets remain OPEN. - PROP|The trusted history epoch is not inferred from queue messages and is not restored from the same backup. Old jobs, outbox rows, parked deliveries, snapshots and writers are fenced. No re-labeling discarded history. - OPEN|SPK-05 must verify real locks, MVCC snapshot semantics, pruning races, connection lifetime, failover, cutover and restore fencing. These browser simulations do not prove PostgreSQL behavior. - OPEN|RPO/RTO remain a business decision (DJ-04). Missing evidence objects are flagged, never silently deleted.