Spike: Spec-based engine updates

[khvn26]

Scope out a way to open issues automatically based on newly added engine tests that are failing, which should open a way for spec-based agent-driven engine updates.

[Zaimwa9]

I'm thinking about using the webhook handler to dispatch the issues creation in the SDK repos. I came up with following.

Pre-requisite in engine-test-data

• Add a pre-commit hook that enforces diff tests contain the description GIVEN / WHEN / THEN
• Add an open-spec-issue action that parses test output to find failing test names, find the tests, extracts the Given/When/Then description, and opens the issue that contains:
  • Title: engine-test-data {release_version}: {count} failing tests
  • Body: one section per failing test that includes the test_name and the GIVEN/WHEN/THEN description and a link to the file

Flow
In the webhook handler, on release/published (or main?) of the engine-test-data:

• dispatches a workflow_dispatch to each SDK that includes the release tag
• Each SDK repo: one new workflow that runs the engine tests and triggers the open-spec-issue from the engine on failure

@khvn26 thoughts, seems doable as a first iteration?

---

Python exemple from Claude

name: Engine spec tests
  on:
    workflow_dispatch:
      inputs:
        tag:
          required: true

  jobs:
    test:
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v4
          with:
            submodules: true

        - run: |
            cd tests/engine_tests/engine-test-data
            git fetch --tags
            git checkout ${{ inputs.tag }}

        - uses: actions/setup-python@v5
          with:
            python-version: "3.12"

        - run: pip install -e ".[dev]"

        - id: run-tests
          run: |
            set -o pipefail
            pytest tests/engine_tests/ --tb=line -q 2>&1 | tee results.txt
          continue-on-error: true

        - if: steps.run-tests.outcome == 'failure'
          uses: Flagsmith/engine-test-data/.github/actions/open-spec-issue@main
          with:
            test-output: results.txt
            tag: ${{ inputs.tag }}

[Zaimwa9]

We've manually released the engine v3.7.0 across edge-api / Python SDK / edge-proxy / flagsmith-engine. While doing it, we setup the first step of the pipeline by opening PRs following the release of a new engine-tests-data (using Renovate).

Based on this, we have PRs opened in all the remaining SDKs that are the basis to integrate the latest change

What's still needed to achieve a Spec driven engine accross the SDKs

• Auto-detect when a Renovate-bumped engine-test-data introduces failing test cases
• An agent that picks up the failing PR, applies the fix, runs tests, pushes a commit
• A "green CI = ready to release" confidence loop with human review at the end (for now!)

Next steps, two possible paths

1. Claude in GitHub: Use the Claude Code GitHub App, trigger via @claude mention or workflow.

• Pros: zero infra, ships today.
• Cons: per-call billing, code/context leaves our infra on each run.

2. Self-hosted agents: runner executing a proprietary model or an OSS model (Devstral, Qwen, DeepSeek, Mistral? 🇫🇷 ), triggered by our existing GitHub webhook handler, running in a sandboxed container, opening PRs via a bot account.

• Pros: cost control at scale, data stays in our infra, fully customizable.
• Cons: ops burden, prompt/tooling work, model quality tradeoff for OSS ?

Possible flow (either path, event-driven)
Renovate bumps submodule → CI fails → GitHub webhook fires (check_run.completed) → handler dispatches an ephemeral worker → agent branches off the Renovate PR, applies the fix referencing the Python engine, pushes → CI green → human review/merge → release.

The worker spins up on demand and exits.
cc @khvn26

[Zaimwa9]

The scope of this issue could be enlarged to include all SDKs development actually, not only engined based (even though it is the most straightforward).
I am thinking about the incoming events tracking that will be ported in all of them.
@khvn26