Skip to content

Reviewer screening rebuild and invalidation

This first Phase 4A slice adds rebuildable membership-screening and reviewer-screening rows, authorized current selectors and shared-root historical observations. Existing consumer endpoints continue to use their authoritative queries. This is a correctness delivery, not a read-performance cutover or completion of incremental Phase 4A maintenance.

Formula compatibility

Both families use ForMembership(InvestigatorId) scopes, including members with no screenings. They remain separate metric families: the membership aggregation uses live agreement measures, while the reviewer query uses persisted InclusionInfo at the configured threshold. Missing persisted inclusion entries need not enter either availability bucket. These answers are not interchangeable.

Membership counters preserve the current value-object mapping, including project-wide sufficiency and overscreening values, the duplicated excluded-agree value in both agree outputs, and availability computed by subtracting reviewer counts from project counts. Percentages remain derived by the existing value object, including its truncation and zero-total rules. Catalogue corrections require a separate deliberate migration; rebuilding never silently corrects legacy formulas.

The rebuild reads Project, source counts and revision controls in one pinned snapshot. Membership uses the production FullStats pipeline; reviewer counts reuse the production persisted-inclusion filters, executed sequentially through the same Mongo session. Removed membership scopes return an authoritative absence rather than a fabricated zero row.

Mutation and rebuild contract

Bounded screening submissions and corrections maintain enabled reviewer families in the source transaction after their canonical rows have been rebuilt. Before classification, a BSON read captures persisted inclusion statuses at the Study revision that the source save compares and replaces. Deserializing ScreeningInfo alone would recompute those statuses and lose legacy parity. Missing or changed source revisions retain fallback. The canonical membership list comes from a Project whose Audit.Version is matched by a conditional write in the source transaction. That write changes a storage-only StatisticsAdmissionToken on the Project without advancing its domain version. A snapshot read alone cannot exclude an independent membership or definition edit; the token write forces a Mongo write conflict if that Project changed after the snapshot, so the source save and its statistics abort together. Concurrent Project replacements also conflict while admission holds the transactional write. The existing source-save conflict/reload path handles retries. The token is not a lease or an externally visible revision, and ordinary Project replacements need not preserve its value. This intentional shared Project write adds contention that future write benchmarks must count.

If all screening family flags turn off after an operation is admitted, Prepare still retains a source-only transaction when project counters changed or reviewer families were captured. That transaction invalidates the relevant families alongside the source write, receipt and notification outbox; abort rolls all of them back. A flag enabled only after admission still cannot publish a reviewer family without its captured epoch. An unchanged project-only operation remains a no-op. This repairs the existing default-off flag contract; it adds no activation switch or outbox protocol.

Reviewer snapshot matching compares the configured nullable agreement ratio exactly, including rejecting NaN as before; nearby floating-point values are different configurations, never a tolerance match. Controller attempt preparation and saving retain per-attempt durable-mode capture, while the operation envelope remains outside the retry loop and domain events dispatch only after successful transactional saves. Family and row admission checks retain their original order and snapshot.

Each family retains its own captured write epoch. Every changed reviewer row must already be published, Fresh and compatible with the current controls, including its family epoch. Selection uses the same bounded lower-generation lookup as the current reader. A point contribution never bootstraps a reviewer row's project-wide totals from zero. A completely classified unchanged family retains its existing state; it is not republished as Fresh. Membership and reviewer flags remain independent of project-screening serving.

Classification considers at most 100 canonical memberships, then obeys the coordinator's stricter 500-move, 100-document and encoded-size ceilings. A boundary transition across 51 memberships can already exceed the document cap because the two families remain distinct. In that case the writer retains any enabled bounded project-screening delta and atomically invalidates reviewer families. Missing baselines, disabled writes, stale epochs, active fences and failed Project version admission also preserve authoritative fallback. Family invalidation advances the write epoch in the source transaction; abort rolls it back, and rebuilding one member cannot revive untouched older rows.

Existing population/import/delete and inclusion-recalculation operations fence all canonical member scopes, then use the registered authoritative family calculators during rebuild. They retain their durable operation ownership and transaction boundaries.

Administrative path and permissions

With writes, the existing membership-screening family flag and the reviewed project allowlist enabled, a batch-project administrator can run:

  • POST /api/admin/project-statistics/{projectId}/membership-screening/rebuild
  • POST /api/admin/project-statistics/{projectId}/reviewer-screening/rebuild

These synchronous triggers reuse existing restartable backfill/checkpoint orchestration and typed contention outcomes. Each enumerates current canonical members, records the observed bootstrap checkpoint, and rebuilds each row. New families use distinct backfill-observed:MembershipScreening:{scope-set-digest} and backfill-observed:ReviewerScreening:{scope-set-digest} triggers, so a project's older screening bootstrap cannot masquerade as their first observation. A repeat for the same exact canonical scope set reuses its own original history point. Adding or removing members changes the scope-set digest and creates a newly observed root; it cannot report a false NoOp using a root that never covered the requested scope set. Earlier immutable roots retain their truthful gaps for scopes they did not observe. Partial work reports incomplete and can be retried. The existing forced-rebuild/retry guidance applies if a different checkpoint occupies the bootstrap identity.

Ordinary current and historical selectors retain their existing graph permissions, active own-row membership requirements, and additional ViewMemberships permission for peer rows. Historical multi-member results resolve one immutable checkpoint root. No cross-root composition is added.

Required next slice before Phase 5.3 activation

The bounded maintenance path is correctness-tested; it does not establish a workload or latency pass. Before moving reviewer consumers to materialized reads, measure this path and the fallback rebuild workload, including large memberships, corrections, resets and population changes. Broader maintenance and rollout acceptance remain in the existing #3255/#3364 work. That work is required for Phase 5.3 activation, not an optional performance improvement. Keep consumer flags disabled until its parity and workload gates pass. Kappa, search families and reporting adapters remain their own planned slices.

Administrative routes

Each reviewer family exposes both an idempotent .../{family}/backfill and a forced .../{family}/rebuild, mirroring the project-screening pair. The distinction is not cosmetic: forcing is the reconciliation that adopts the authoritative configuration identity inside the publication's own control compare-and-swap and republishes every scope, so it must not be a family's only entry point. The non-forced pass is the ordinary operational trigger.

Administrative responses identify the selected metric family and expose the same typed 202 summary, 409 problem details and 404 responses in the generated client. A repeated non-forced backfill selects unchanged lower-generation rows exactly as the current reader does — the exact row at the guard's visible generation when it exists, otherwise the greatest published row below it — so rebuilding multiple reviewers does not churn a revision on every repeat. A publication that did not happen is reported with the disposition that matches what actually happened to the row: Stale when the rebuild ran and returned the scope to Stale, Absent when the family says the scope does not exist, and Contended when the rebuild never started — another owner holds the live lease or the guard's sole candidate slot, or a fence covered the family before a candidate was claimed — so the published row is untouched and still serving. All three are partial failures that answer 409, but only Stale means the projection degraded, and ScopesStale is the number an operator reads to decide whether to escalate. Lease contention does not imply that an existing Fresh row was changed.

Deferred to the read-side parity slice

Per-family IProjectStatisticsAuthoritativeCalculator implementations for the two reviewer families are deliberately not in this slice. That contract's only consumer is ProjectScreeningParityAuditService, which takes exactly one calculator, so wiring reviewer families into shadow parity needs a per-family audit consumer — design work belonging with the reviewer consumer cutover, not with rebuild and history. Adding unregistered implementations ahead of it would be code nothing resolves and nothing calls.

Sweep completion under concurrent writes

A failed scope rebuild marks the row selected by the reader non-servable, including a published row below the family's latest generation. A later successful scope cannot restore that failed row. After all scopes complete, the sweep verifies their rows, epochs, versions and configuration against the controls and publication guard in one pinned snapshot. If a concurrent write made an earlier row incompatible, the administrative call returns a retryable 409 even when each scope published successfully in isolation. Published-scope counts still describe the work performed; they do not override the failed final verification. Cancellation reaches the MongoDB aggregation so an abandoned backfill can release its source snapshot promptly.

The final check also re-enumerates the family's canonical scopes using the verifier's pinned snapshot and session, so membership and projection checks describe the same database boundary. A member added during the sweep makes completion fail until retry includes that member; per-scope publication checks alone cannot prove a frozen scope list is complete. Forced rebuilds skip the preliminary already-current calculation and leave authoritative computation to the bootstrap and lifecycle publication paths.