compiles valuable GUI trajectories into parameterized, self-verifying CLI macros for agents#226
Conversation
Implements the full 7-layer architecture described in the OpenClaw Macro System spec as a working CLI-Anything agent harness: - cli-anything-openclaw CLI (L6 Unified CLI Entry) macro run/list/info/validate/dry-run/define session status/history/save/list backends (shows routing availability) Interactive REPL (default) - MacroRuntime (L5): validate params -> check preconditions -> execute steps -> check postconditions -> collect outputs -> telemetry - MacroDefinition + YAML loader (L4): parameterized, composable, self-verifying macro definitions stored in macro_definitions/ - RoutingEngine (L3): selects backend by priority (native_api=100, gui_macro=80, file_transform=70, semantic_ui=50, recovery=10) - Execution Backends (L2): native_api -- subprocess / shell commands file_transform -- XML, JSON, text file editing semantic_ui -- accessibility + keyboard shortcuts (xdotool on Linux) gui_macro -- precompiled coordinate-based replay (pyautogui) recovery -- retry + fallback orchestration - ExecutionSession: run history, telemetry, disk persistence 64/64 tests passing (test_core.py + test_full_e2e.py with real CLI subprocess)
|
Thanks for this PR. This direction is interesting. And it's even qualified to be another independent project/repo, considering the scope. What this seems to add is less a traditional single-software harness and more a general agent-callable macro runtime for turning reusable GUI workflows into parameterized CLI macros. I think Local verification on this branch was good:
I did not find an obvious serious bug in the implementation. A couple of non-blocking things still seem worth clarifying:
Overall though, the implementation and tests look solid, and I think this is mergeable. |
|
Thank you for your recognition! I’ve noticed that CLI alone has difficulty handling certain closed-source software or apps. With that in mind, I proposed this PR based on the idea of combining CLI and GUI to build a more general-purpose agent. The goal is to provide a simple way to orchestrate GUI operations and convert them into something callable via CLI, which would better align with the name “cli-anything.” However, this PR is not yet fully developed. At the moment, GUI orchestration still relies on manual effort, and there are likely many unresolved bugs. I’m currently working on improving these capabilities and will address the issues you’ve raised. I hope to have your support so I can continue refining this through future PRs and resolve the remaining problems. If anyone else finds this idea valuable, they’re very welcome to contribute to its improvement and development. Many thanks! |
Description
Adds the OpenClaw Macro System — a layered CLI that converts GUI workflows
into parameterized, agent-callable macros. Agents invoke macros through a stable
CLI (
macro run <name> --param k=v --json); the system handles backend routingacross native APIs, file transforms, accessibility controls, and compiled GUI
replay — transparent to the caller.
Type of Change
For New Software CLIs (in-repo)
<SOFTWARE>.mdSOP document exists atopenclaw-skill/agent-harness/OPENCLAW.mdSKILL.mdexists inside the Python package (cli_anything/openclaw/skills/SKILL.md)cli_anything/openclaw/tests/test_core.pyare present and pass without backendcli_anything/openclaw/tests/test_full_e2e.pyare presentregistry.jsonincludes an entry withsource_url: nullrepl_skin.pyinutils/is an unmodified copy from the pluginGeneral Checklist
--jsonflag is supported on all commandsfeat:)Test Results
platform linux -- Python 3.11.15, pytest-9.0.3
collected 64 items
tests/test_core.py::TestMacroModel::test_load_from_yaml PASSED
tests/test_core.py::TestMacroModel::test_validate_params_required PASSED
tests/test_core.py::TestMacroModel::test_validate_params_type_error PASSED
tests/test_core.py::TestMacroModel::test_validate_params_range PASSED
tests/test_core.py::TestMacroModel::test_validate_params_ok PASSED
tests/test_core.py::TestMacroModel::test_resolve_params_defaults PASSED
tests/test_core.py::TestMacroModel::test_structural_validation_no_steps PASSED
tests/test_core.py::TestMacroModel::test_structural_validation_bad_backend PASSED
tests/test_core.py::TestMacroModel::test_to_dict PASSED
... (49 unit tests)
tests/test_full_e2e.py::TestCLISubprocess::test_macro_run_json_transform_workflow PASSED
tests/test_full_e2e.py::TestCLISubprocess::test_unknown_macro_returns_error_json PASSED
... (15 E2E tests)
64 passed in 0.80s