Rigorous, reproducible derived-parquet pipeline + AI-free tests (#273)

[rdhyee]

Implements the rigorous, reproducible, AI-free-tested data pipeline tracked in #273.

What this delivers (all human-runnable: make test / make all)

• scripts/build_frontend_derived.py (hardened rewrite): geometry-agnostic (WKB BLOB or DuckDB GEOMETRY); decorrelated concept resolution; first-non-root material (Issues with material types #265/build: stop SKOS root 'Material' leaking into the material facet (#265) #271); deterministic COPY ordering + tie-broken dominant_source; hard-fails on duplicate pids / concept row-ids; emits {tag}_manifest.json (input+output sha256, argv, git SHA, DuckDB+extension versions).
• scripts/validate_frontend_derived.py — a semantic trust gate, not a spot check: asserts the derived-file algebra (facet_summaries == GROUP BY facets, facet_cross_filter == conditional GROUP BY, facets.pid == map_lite.pid, pid uniqueness, H3 sums, scheme), and with --wide re-derives from the source wide and diffs the written facets/map_lite/h3 (cells, counts, dominant_source, resolution, tolerant centers). Non-zero exit on any failure.
• tests/test_frontend_derived.py — 12 fixture tests (no network/big data): both geometry encodings, material/concept/place_name cases, dup-pid hard-fail, manifest, wide_h3 correctness, and two tests that prove the gate catches corruption internal checks miss.
• Makefile, scripts/requirements.txt (duckdb==1.4.4), .github/workflows/pipeline-tests.yml (CI fixture gate).
• DATA_PROVENANCE.md + SERIALIZATIONS.md reconciled with reality.

Proven by execution (not by review)

• Builder: 5.4 s on the real 20M-row 202604 wide (the prior approach was a >16-min perf blowup, killed).
• --wide semantic gate: exits 0 on the real 202604 rebuild; exits 1 when data is corrupted (a test zeroes coordinates — passes internal checks, fails the gate).
• Two-run determinism: facets / map_lite / summaries / cross_filter are byte-identical across runs.
• The validator run against live prod correctly fails (346,768 root rows) — the bug is real and the gate detects it.

Process

Scope + correctness hardened by three adversarial Codex rounds — Codex literally proved an earlier validator passed a wrecked rebuild; that hole (and the follow-ups: h3 resolution/centers, scheme, wide_h3) is closed and each fix has a test. AI sign-off was not the gate — the executable tests are.

Threat-model note: the --wide gate imports the same build_base_tables, so it catches corruption / staleness / wrong-version of published artifacts; builder-logic correctness is covered separately by the fixture tests (which assert against hand-written expected values).

Notes

• ⚠️ Stacks on docs+scripts: data provenance map + build scripts for the 6 unscripted derived parquet #264 (introduces build_frontend_derived.py); diff collapses once docs+scripts: data provenance map + build scripts for the 6 unscripted derived parquet #264 merges.
• Supersedes build: stop SKOS root 'Material' leaking into the material facet (#265) #271 — its material fix is folded in, correctly and performantly.
• Does NOT publish to R2. The rebuilt 202606 files + manifest are generated and validated locally; the production publish (R2 upload + current/manifest.json cutover) is left for a human gate.

Closes #273 (pending the publish step, tracked there).

---

— 🤖 rbotyee (RY directing, out-of-office autonomous build). Codex adversarial ×3.

[rdhyee]

Adversarial proof round (workflow: independent re-exec + attack + verdict)

Ran a workflow that independently re-executes the pipeline and attacks it. It found two real misses, both now fixed + regression-tested (re-running the exact attacks against the fixed gate now exits 1; the clean rebuild still exits 0):

1. H3 centroids within tolerance — shifting every cell center ~9m (8e-5°) passed the loose 1e-4 tolerance. → tightened to 1e-5 (~1m); residual undetected error bounded at ~1m on display-only centroids.
2. manifest.json never validated — corrupting its sha256 attestations passed. → validator now verifies every output file's sha256 (and the input's, with --wide) against the manifest.

Honest verdict (NOT "foolproof")

Proven (by execution, on the real 292MB wide): no root-material leak, sentinel preserved, pid uniqueness + set-equality, summaries == GROUP BY facets, cross_filter == conditional GROUP BY, H3 sums + discrete cells + centroids, schema, manifest integrity, byte-identical re-runs (discrete files).

Documented boundaries (not bugs — scope):

• The --wide gate imports the same build_base_tables, so it catches corruption/staleness/wrong-version but not builder-logic bugs — those are covered only by the fixture tests' hand-written expectations.
• The manifest is self-attesting, not signed — catches corruption that didn't also rewrite the manifest, not a consistent file+manifest rewrite.
• The deployed 202601 files are not reproducible (unrecorded pipeline).
• The R2 publish + current/manifest.json cutover is human-gated (not automated).

Use it with confidence for gross-error + build-vs-deploy-divergence detection; it is not a complete tamper/provenance-attestation system without signing + automated publish.