RFC: Signed receipts for computer-use agent actions

[tomjwxf]

Problem

CUA enables computer-use agents that interact with desktop applications. When agents click, type, and navigate GUIs autonomously, there is no cryptographic audit trail of what actions were taken, what the agent observed, and what policy governed the decisions. For enterprise deployments of computer-use agents, this is a compliance requirement.

Proposal

Add Ed25519 receipt signing to CUA's action execution layer. Each GUI action (click, type, scroll) would produce a signed receipt capturing: action type, target element hash, screenshot hash (before/after), policy evaluation, and timestamp.

Reference

protect-mcp (MIT, v0.5.3). Receipt format: IETF Internet-Draft.

Happy to discuss and contribute.

[m13v]

the ed25519 receipt signing proposal is solid but i think the bigger practical challenge is what you hash for the target element. pixel coordinates change constantly, even between sequential screenshots of the same screen (rendering jitter, cursor blink, notification popups). we found that hashing the accessibility tree path to the target element (role + name + position in tree) gives you a stable identifier that survives visual noise. for the before/after screenshot hashes, consider hashing a cropped region around the target rather than the full screen. full screenshots produce different hashes even when the actual interaction area is identical, which creates noisy audit logs that are hard to review. the policy evaluation field is the most valuable part of this proposal, honestly. being able to prove that the agent checked its constraints before acting is what compliance teams actually want to see.

[m13v]

our element interaction layer that identifies targets by accessibility tree path rather than coordinates, which might be useful for stable receipt hashing: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs - the element struct carries role, name, and tree position which gives you a stable hash target across visual changes.

[tomjwxf]

@m13v both points are spot on. Built a small proof of concept that incorporates your feedback:

https://gist.github.com/tomjwxf/c14b9b12a28d61837a90ba7bc8ff162d

Uses accessibility tree path (role, name, tree_position) for stable element hashing instead of pixel coordinates, exactly like the terminator element struct you linked. Cropped region hashing around the target element bounds instead of full screenshots. Receipt chain is observation -> policy_check -> execution, each signed and parent-chained.

python3 cua_receipts.py

--- Audit trail ---
  [observation] rcpt-34828c4e...
    Saw: button:Submit at Window:Login > Form:LoginForm > Button:Submit
  [policy_check] rcpt-fcbd96a7...
    Policy: permit_form_submission -> ALLOW
  [execution] rcpt-d51d55ba...
    Action: left_click -> executed

Standalone, no dependencies beyond PyNaCl for Ed25519 (falls back to HMAC for demo). Roughly 200 lines. Figured working code is more useful than more words.

Happy to adapt this to hook into the terminator element layer directly if it's useful.

[m13v]

this is great. using the accessibility tree path tuple is exactly the right call, way more stable than pixel coords across resolution changes and UI reflows. the cropped region hashing is a smart optimization too, full screenshot hashes would invalidate on any unrelated UI change.

one thing to watch for: some apps dynamically reorder siblings in the accessibility tree (especially web apps with virtual scrolling), so you might want to include a content hash fallback for cases where tree_position shifts but the element is clearly the same node. receipt chain looks clean.

[tomjwxf]

Good catch, the virtual-scrolling and sibling-reorder case is real. Web apps with virtualized lists in particular will rewrite tree_position while keeping the logical element stable, and any receipt scheme that treats tree_position as the only binding will fire false-positive drift alerts on those apps.

Plan for v0.2:

1. Add a content-hash fallback channel: content_hash = sha256(role || normalize(name) || stable_attrs) where stable_attrs is an explicit whitelist (e.g. aria-label, placeholder, href, and not position-derived attributes). Receipts carry both tree_path and content_hash.

2. Verifier resolves in two stages: if tree_path matches, done. Otherwise fall back to content_hash with a flag on the receipt recording that a reorder was tolerated. That preserves the audit trail when legitimate reorders happen but still makes the tolerance visible for review.

3. For apps where the accessibility tree is outright missing or lying (some Electron shells, some game engines), the fallback stops and the receipt records content_hash_only as the binding mode. Callers can decide whether to accept that tier for their use case.

Will iterate on the gist and open a proper repo once the design stabilizes. Thanks again for the terminator pointer, the element struct layout in crates/terminator/src/element.rs is a good reference for the stable-hash side.