# Projection bootstrap and restore Nonfinal architecture review · 0.4.0. Generated from `assets/fixtures-architecture.js`; edit that source and rerun `node tools/generate-docs.js`. Proposed, awaiting acceptance and PostgreSQL spike SPK-05. Leases use server time and fail closed on expiry. All exported state and its cursor must come from the same primary transaction snapshot; never route these reads to a lagging replica. The retention-control lock serializes pin creation, advancement and pruning. ## protocol - Serialize writers - Lock the cursor row for tenant_id + catalog_id + history_epoch BEFORE changing exported state. Update the transactional counter, accepted rows and outbox in one transaction. Rollback consumes no position. A delayed writer blocks later commits in this stream. - PROP - Pin retention - Under the retention-control lock shared by the pruner, create a renewable rebuild lease pinned after current pruned-through cursor P. Commit before opening the snapshot. - PROP - Read one snapshot - REPEATABLE READ READ ONLY: read history_epoch, committed stream cursor W, and all accepted state including private/public releases and aggregate revisions in one snapshot. Never use max(outbox.sequence) or a PostgreSQL sequence. - PROP - Publish export - Advance lease pin from P to W under retention coordination. Write checksummed export objects. Manifest becomes ready only after export completion and a valid lease check; abort on epoch change. - PROP - Load shadow - Load the manifest into an isolated projection generation, with aggregate revisions and W as its starting contiguous applied position. Renew the lease throughout export/load/catch-up. - PROP - Retrieve retained events - GET /v1/exports/events?after=W&lease_id=…&history_epoch=… returns ordered authorized stream events. The pruner cannot remove events after an active pin. A stale cursor or expired lease returns 410; restart from a new export. - PROP - Catch up and cut over - Choose committed target H. Apply every position through H, retaining dedupe, revisions and dependency parking. Require no unresolved dependencies at/below H. Pause the local applier, atomically swap the serving generation and durable consumer cursor, resume after H, then release the lease. - PROP - Advance live retention progress - Commit projection changes, inbox outcomes, dependency parking and the greatest contiguous processed position together in the live product transaction. Persist/report progress monotonically to resolver retention control for the same tenant, catalog and history epoch only after that commit; an interrupted report conservatively retains extra events. The pruner may use only acknowledged durable progress. Shadow catch-up has a separate cursor and lease; it never advances the live cursor. Cutover atomically swaps serving generation and durable live cursor before releasing the rebuild lease. - PROP - Fence restored history - Pause writers, relay, consumers and public serving; issue a NEW epoch in a trusted control plane outside the restored database. Fence old workers; abandon old events without relabeling; initialize a new cursor and rebuild from restored state. Audit and terminally ignore old-epoch deliveries. - PROP ## retention ### data Evidence artifacts (HTML, JSON, screenshots) ### store Private evidence bucket ### retention Kept while referenced by accepted facts, decisions, open cases or regression fixtures; unreferenced expire after a window (DEC-06) ### status OPEN ### recovery Object-store durability + versioning/replication choice OPEN; not rebuildable ### rpo OPEN ### rto OPEN ### data Claims, extraction runs, unmapped fields ### store resolver schema ### retention As long as the artifact they cite (PROP) ### status PROP ### recovery Database PITR; also re-derivable by replay from retained artifacts ### rpo Same as resolver DB ### rto Same as resolver DB ### data Accepted state (items, mappings, policies) ### store resolver schema ### retention Permanent (current + revisions) ### status PROP ### recovery Database PITR ### rpo OPEN (DJ-04) ### rto OPEN (DJ-04) ### data Decision history + review cases ### store resolver schema ### retention Permanent; immutable (DJ-08) ### status PROP ### recovery Database PITR ### rpo OPEN (DJ-04) ### rto OPEN (DJ-04) ### data Integration events (outbox) ### store resolver.outbox ### retention Prune only a contiguous prefix of published events whose retention age has elapsed, through the minimum of every registered live consumer’s durable contiguous progress and every unexpired rebuild lease pin. Serialize lease creation/advancement, consumer registration and pruning with the retention-control lock. Delivery or acknowledgement alone is not application progress; a missing position or parked dependency protects the tail. Monitor blocked retention and lease expiry; never silently remove a lagging live consumer. ### status PROP ### recovery Not backed up separately; lost tail is recovered by snapshot rebuild ### rpo n/a ### rto n/a ### data Product inbox + parked events ### store product schema ### retention At least the outbox window so late duplicates are still caught ### status PROP ### recovery Rebuilt with the projection ### rpo n/a ### rto n/a ### data Product projections + search docs ### store product schema / index ### retention Current only ### status PROP ### recovery Rebuild from snapshot + tail (ADR-004); backups optional ### rpo n/a (derived) ### rto Rebuild time — to measure (SPK-05) ### data Snapshot exports ### store Object storage ### retention Keep the last few successful exports (count OPEN) ### status OPEN ### recovery Re-export ### rpo n/a ### rto n/a ### data Database backups ### store Managed provider PITR ### retention Provider window, chosen with DJ-04 ### status OPEN ### recovery Restore while stopped; rotate externally trusted history epoch; rebuild projections; discard old-epoch deliveries ### rpo OPEN ### rto OPEN ## risks - PROP|Every projection-affecting write uses the cursor lock; unrelated evidence intake need not. Per-stream serialization is a throughput tradeoff to measure. - PROP|Pruning respects active rebuild pins AND the durable live-consumer cursor. Expired leases cannot be revived; a rebuild restarts. Lease duration and retained-byte budgets remain OPEN. - PROP|The trusted history epoch is not inferred from queue messages and is not restored from the same backup. Old jobs, outbox rows, parked deliveries, snapshots and writers are fenced. No re-labeling discarded history. - OPEN|SPK-05 must verify real locks, MVCC snapshot semantics, pruning races, connection lifetime, failover, cutover and restore fencing. These browser simulations do not prove PostgreSQL behavior. - OPEN|RPO/RTO remain a business decision (DJ-04). Missing evidence objects are flagged, never silently deleted. ## adr ### id ADR-004 ### slug projection-bootstrap-and-recovery ### title Transactional cursor, pinned rebuild and restored-history fence ### status Proposed ### br - BR-03 ### context Product projections must be rebuildable (new deployments, schema changes, corruption) without making the outbox a permanent log. ### decision Lock the cursor row for tenant_id + catalog_id + history_epoch BEFORE changing exported state. Update the transactional counter, accepted rows and outbox in one transaction. Rollback consumes no position. A delayed writer blocks later commits in this stream. Under the retention-control lock shared by the pruner, create a renewable rebuild lease pinned after current pruned-through cursor P. Commit before opening the snapshot. REPEATABLE READ READ ONLY: read history_epoch, committed stream cursor W, and all accepted state including private/public releases and aggregate revisions in one snapshot. Never use max(outbox.sequence) or a PostgreSQL sequence. Advance lease pin from P to W under retention coordination. Write checksummed export objects. Manifest becomes ready only after export completion and a valid lease check; abort on epoch change. Load the manifest into an isolated projection generation, with aggregate revisions and W as its starting contiguous applied position. Renew the lease throughout export/load/catch-up. GET /v1/exports/events?after=W&lease_id=…&history_epoch=… returns ordered authorized stream events. The pruner cannot remove events after an active pin. A stale cursor or expired lease returns 410; restart from a new export. Choose committed target H. Apply every position through H, retaining dedupe, revisions and dependency parking. Require no unresolved dependencies at/below H. Pause the local applier, atomically swap the serving generation and durable consumer cursor, resume after H, then release the lease. Commit projection changes, inbox outcomes, dependency parking and the greatest contiguous processed position together in the live product transaction. Persist/report progress monotonically to resolver retention control for the same tenant, catalog and history epoch only after that commit; an interrupted report conservatively retains extra events. The pruner may use only acknowledged durable progress. Shadow catch-up has a separate cursor and lease; it never advances the live cursor. Cutover atomically swaps serving generation and durable live cursor before releasing the rebuild lease. Pause writers, relay, consumers and public serving; issue a NEW epoch in a trusted control plane outside the restored database. Fence old workers; abandon old events without relabeling; initialize a new cursor and rebuild from restored state. Audit and terminally ignore old-epoch deliveries. Prune only a contiguous prefix of published events whose retention age has elapsed, through the minimum of every registered live consumer’s durable contiguous progress and every unexpired rebuild lease pin. Serialize lease creation/advancement, consumer registration and pruning with the retention-control lock. Delivery or acknowledgement alone is not application progress; a missing position or parked dependency protects the tail. Monitor blocked retention and lease expiry; never silently remove a lagging live consumer. ### consequences - Writer serialization limits throughput per tenant/catalog stream - Active leases and consumer lag can extend retention; monitor and budget - Restore requires coordinated downtime and an external epoch - SPK-05 remains necessary ### alternatives - Coordinated write-quiescence snapshot boundary - Transactional change-data-capture cursor ### evidence SPK-05 shows divergence or unacceptable rebuild time.