capture: add multi-display observability and adaptive OCR budgets

[haasonsaas]

Summary

Improve capture quality and cost control with stable display identifiers, per-display telemetry, and adaptive OCR budgets driven by backlog and duplicate/drop behavior.

Why

agentd currently captures the first display and uses one global cadence. That is fine for v0, but dogfood will quickly hit multi-monitor setups, retina/non-retina differences, display changes, high OCR cost, and noisy windows. We need observability before blindly increasing capture volume.

SOTA notes

• ScreenCaptureKit samples emphasize explicit content filters, configurable stream output, and bounded queue depth.
• Apple's sample notes queue depth affects memory and should stay bounded; agentd already keeps queueDepth = 5, which is good.
• Screenpipe's active backlog includes per-monitor observability and stable display IDs, which maps directly to this product surface.

Acceptance

• Record stable display metadata in frame/batch metadata where possible: display id, size, scale, active/main display state.
• Heartbeat or diagnostics report includes per-display capture counts, drops, OCR latency estimate, and backpressure drops.
• Handle display attach/detach without requiring app restart.
• Add config/policy for selected displays or capture-all-displays, with conservative defaults.
• Add adaptive OCR/capture budget behavior when backlog, OCR latency, or droppedBackpressure exceed thresholds.
• Tests cover adaptive budget state transitions and metadata encoding; manual smoke covers multi-display behavior.

References

• Apple ScreenCaptureKit sample docs: content filters and bounded queue depth.
• Comparable SOTA signal: vision capture: per-monitor observability + stable display IDs (follow-ups from #3017) screenpipe/screenpipe#3029 tracks per-monitor observability and stable display IDs.

[haasonsaas]

Progress update: PR #40 merged in 1131848.

What landed for this issue:

• Frame payloads now include display id, display scale, and main-display metadata.
• Diagnostics reports expose queue pressure and batch state without raw OCR.
• Adaptive OCR persistence budgets reduce stored OCR text under backlog/backpressure while still scanning full OCR text for secrets before truncation.
• Tests cover display metadata encoding and adaptive OCR budget transitions.

Remaining before this issue should close: true capture-all/multi-display concurrent capture, display attach/detach handling without restart, selected-display policy/config, and a manual multi-display smoke pass.

[haasonsaas]

Progress update: PR #48 merged in c47f483 / squash on 2026-04-29.

Follow-up fixed from the backlog sweep: Chronicle heartbeat now reports only in-memory pending frames in pendingFrameCount while still including local queued-batch bytes in pendingBytes. That removes the batch-file-count/frame-count conflation from the multi-display/observability path.

Remaining scope on this issue is still the larger capture work: true capture-all/multi-display concurrency, attach/detach handling without restart, selected-display policy/config, and a manual multi-display smoke pass.

[haasonsaas]

Progress update: PR #49 merged in e95c602 on 2026-04-29.

This moves the multi-display backlog materially forward:

• CaptureService now starts one SCStream per configured display instead of always using the first display.
• captureAllDisplays and selectedDisplayIds are supported in local config and managed CapturePolicy; capture restarts when policy changes the display scope.
• Diagnostics now include per-display size, scale, main-display marker, enqueued-frame count, dropped-frame count, and last-frame timestamp.
• RegisterDevice metadata now includes capture display scope.
• Default remains conservative: one display unless config/policy opts into all/selected displays.

Validation: swift test, warnings-as-errors build, Swift format lint, mock Chronicle fixture validation, shellcheck, package build, permission smoke --no-launch, Codex Run ./script/build_and_run.sh --verify, Computer Use inspection of System Settings Accessibility, and log evidence from a desktop launch.

Remaining before closing: real multi-monitor hardware smoke with attached/detached display behavior and any server-side fan-in/audit UX follow-up.

[haasonsaas]

Progress update: PR #81 merged on 2026-04-29.

Added scripts/multi_display_smoke.sh so the remaining multi-monitor pass is reproducible and bounded:

• records agentd list-displays JSON snapshots by phase (before-attach, after-attach, capture-all, after-detach);
• appends a Markdown report without screenshots or OCR text;
• supports --require-multiple, which exits 78 when an attached-display phase still sees fewer than two displays;
• documents the helper in README next to the other local smoke commands.

Local baseline from this machine: one display visible (Built-in Retina Display, display id 1, 1800x1169), Accessibility trusted, Screen Capture trusted. The deliberate --require-multiple after-attach probe failed with exit 78 because this session has only one display, so this still should not close #34.

Validation before merge: shellcheck, helper --help, baseline smoke report generation, expected --require-multiple failure on one display, swift test, warnings-as-errors build, Swift format lint, mock Chronicle fixtures, Chronicle parity strict audit, and git diff --check. CI and Socket passed on #81; Bugbot left no review threads.

Remaining before close: run the same helper on real multi-monitor hardware across attach/capture-all/detach phases and attach the generated report plus sanitized diagnostics/batch metadata counts.