Skip to content

compiles valuable GUI trajectories into parameterized, self-verifying CLI macros for agents#226

Merged
yuh-yang merged 2 commits intoHKUDS:mainfrom
haorui-harry:main
Apr 18, 2026
Merged

compiles valuable GUI trajectories into parameterized, self-verifying CLI macros for agents#226
yuh-yang merged 2 commits intoHKUDS:mainfrom
haorui-harry:main

Conversation

@haorui-harry
Copy link
Copy Markdown
Contributor

@haorui-harry haorui-harry commented Apr 17, 2026

Description

Adds the OpenClaw Macro System — a layered CLI that converts GUI workflows
into parameterized, agent-callable macros. Agents invoke macros through a stable
CLI (macro run <name> --param k=v --json); the system handles backend routing
across native APIs, file transforms, accessibility controls, and compiled GUI
replay — transparent to the caller.

Type of Change

  • New Software CLI (in-repo) — adds a CLI harness inside this monorepo

For New Software CLIs (in-repo)

  • <SOFTWARE>.md SOP document exists at openclaw-skill/agent-harness/OPENCLAW.md
  • SKILL.md exists inside the Python package (cli_anything/openclaw/skills/SKILL.md)
  • Unit tests at cli_anything/openclaw/tests/test_core.py are present and pass without backend
  • E2E tests at cli_anything/openclaw/tests/test_full_e2e.py are present
  • registry.json includes an entry with source_url: null
  • repl_skin.py in utils/ is an unmodified copy from the plugin

General Checklist

  • Code follows existing patterns and conventions
  • --json flag is supported on all commands
  • Commit messages follow the conventional format (feat:)
  • Changes tested locally

Test Results

platform linux -- Python 3.11.15, pytest-9.0.3
collected 64 items

tests/test_core.py::TestMacroModel::test_load_from_yaml PASSED
tests/test_core.py::TestMacroModel::test_validate_params_required PASSED
tests/test_core.py::TestMacroModel::test_validate_params_type_error PASSED
tests/test_core.py::TestMacroModel::test_validate_params_range PASSED
tests/test_core.py::TestMacroModel::test_validate_params_ok PASSED
tests/test_core.py::TestMacroModel::test_resolve_params_defaults PASSED
tests/test_core.py::TestMacroModel::test_structural_validation_no_steps PASSED
tests/test_core.py::TestMacroModel::test_structural_validation_bad_backend PASSED
tests/test_core.py::TestMacroModel::test_to_dict PASSED
... (49 unit tests)
tests/test_full_e2e.py::TestCLISubprocess::test_macro_run_json_transform_workflow PASSED
tests/test_full_e2e.py::TestCLISubprocess::test_unknown_macro_returns_error_json PASSED
... (15 E2E tests)

64 passed in 0.80s

Implements the full 7-layer architecture described in the OpenClaw Macro
System spec as a working CLI-Anything agent harness:

- cli-anything-openclaw CLI (L6 Unified CLI Entry)
  macro run/list/info/validate/dry-run/define
  session status/history/save/list
  backends (shows routing availability)
  Interactive REPL (default)

- MacroRuntime (L5): validate params -> check preconditions ->
  execute steps -> check postconditions -> collect outputs -> telemetry

- MacroDefinition + YAML loader (L4): parameterized, composable,
  self-verifying macro definitions stored in macro_definitions/

- RoutingEngine (L3): selects backend by priority (native_api=100,
  gui_macro=80, file_transform=70, semantic_ui=50, recovery=10)

- Execution Backends (L2):
  native_api    -- subprocess / shell commands
  file_transform -- XML, JSON, text file editing
  semantic_ui   -- accessibility + keyboard shortcuts (xdotool on Linux)
  gui_macro     -- precompiled coordinate-based replay (pyautogui)
  recovery      -- retry + fallback orchestration

- ExecutionSession: run history, telemetry, disk persistence

64/64 tests passing (test_core.py + test_full_e2e.py with real CLI subprocess)
@yuh-yang yuh-yang added the community-contrib-CLI Community-contributed CLI or harness label Apr 17, 2026
@yuh-yang
Copy link
Copy Markdown
Collaborator

Thanks for this PR. This direction is interesting. And it's even qualified to be another independent project/repo, considering the scope.

What this seems to add is less a traditional single-software harness and more a general agent-callable macro runtime for turning reusable GUI workflows into parameterized CLI macros. I think
that is a valuable direction for CLI-Anything.

Local verification on this branch was good:

  • python3 -m pytest cli_anything/openclaw/tests/test_core.py -v -> 49 passed
  • python3 -m pytest cli_anything/openclaw/tests/test_full_e2e.py -v -> 15 passed

I did not find an obvious serious bug in the implementation.

A couple of non-blocking things still seem worth clarifying:

  1. Since this is more of a macro/runtime layer than a standard app-specific harness, the naming and positioning could be explained a bit more clearly in the PR / docs.
  2. The registry.json homepage probably should not point back to the CLI-Anything repo. You could submit a PR later if you have plan to make this a separate project.

Overall though, the implementation and tests look solid, and I think this is mergeable.

@yuh-yang yuh-yang merged commit d3471a8 into HKUDS:main Apr 18, 2026
@haorui-harry
Copy link
Copy Markdown
Contributor Author

Thank you for your recognition! I’ve noticed that CLI alone has difficulty handling certain closed-source software or apps. With that in mind, I proposed this PR based on the idea of combining CLI and GUI to build a more general-purpose agent. The goal is to provide a simple way to orchestrate GUI operations and convert them into something callable via CLI, which would better align with the name “cli-anything.”

However, this PR is not yet fully developed. At the moment, GUI orchestration still relies on manual effort, and there are likely many unresolved bugs. I’m currently working on improving these capabilities and will address the issues you’ve raised.

I hope to have your support so I can continue refining this through future PRs and resolve the remaining problems. If anyone else finds this idea valuable, they’re very welcome to contribute to its improvement and development. Many thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contrib-CLI Community-contributed CLI or harness

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants