Skip to content

test(awk): add rewritten oracle harness#233

Open
matt-dz wants to merge 18 commits into
mainfrom
codex/awk-bwk-tests
Open

test(awk): add rewritten oracle harness#233
matt-dz wants to merge 18 commits into
mainfrom
codex/awk-bwk-tests

Conversation

@matt-dz
Copy link
Copy Markdown
Collaborator

@matt-dz matt-dz commented May 6, 2026

Summary

Adds the rewritten AWK scenario harness and pinned GNU awk 5.4.0 oracle that future rshell awk implementation PRs build on. The harness keeps implementation gating opt-in through enabled.txt, runs rshell through an awk-shaped adapter, and preserves upstream provenance without requiring live upstream test fetches.

  • Adds local rewritten AWK scenarios plus an upstream mapping ledger.
  • Adds tools/awk-harness with pinned GNU awk install support and the rshell-awk adapter.
  • Wires Make/CI entry points for metadata checks and rewritten scenario runs.
  • Adds the implement-awk skill and compatibility guidance for future work.

Validation

  • bash -n tools/awk-harness/*.sh tools/awk-harness/rshell-awk
  • go test ./tests -run TestAwkScenarioMetadata -count=1
  • make test_awk_rewritten

@matt-dz matt-dz force-pushed the codex/awk-bwk-tests branch 3 times, most recently from 558c2bb to 6cd7980 Compare May 6, 2026 17:56
@matt-dz matt-dz marked this pull request as ready for review May 6, 2026 18:40
@matt-dz matt-dz enabled auto-merge May 6, 2026 18:43
@@ -0,0 +1,155 @@
---
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Topic from slack:

I'm wondering if could just compare to mawk already available in debian:bookworm-slim that we already use:

docker run --rm debian:bookworm-slim bash -c 'awk -W version'
random-funcs:       srandom/random
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer     2147483647
mawk 1.3.4 20200120
Copyright 2008-2019,2020, Thomas E. Dickey
Copyright 1991-1996,2014, Michael D. Brennan

Copy link
Copy Markdown
Member

@AlexandreYang AlexandreYang May 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also:
I think it would be nice to make it work for scenarios tests "bash" comparison mode
which means possible installing gawk in debian:bookworm-slim or another image

@@ -0,0 +1,155 @@
---
name: implement-awk
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Topic 2 from slack:

I'm not fully sure if we should just use the upstream tests suites as it,

we might have/want behaviour that slight differ from upstream awk
features that we don't implement
features we differ on purpose


also, beside the previous point, I believe, it's not ideal I believe to have this kind of dependency on upstream repo:

  • slower CI
  • less predictability, more flaky
  • more complex setup

the good thing about replicating the tests as our own scenario tests is that:

  • we have full control for divergent behaviour
  • we can easily add more tests (if we use upstream, we can add our own scenarios tests too, but it's harder to track wasn't missing or not, and have 2 sources of tests)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, agreed that we shouldn’t run the upstream suites as-is or make them the main CI contract.

We do have harness filters, though, so I think the intended use is a curated allowlist/smoke profile: run only known-relevant upstream cases for the rshell awk subset, then translate any important discoveries into first-class tests/scenarios/ coverage.

So the split I’d suggest is:

  • tests/scenarios/ stay authoritative for normal CI and intentional rshell behavior.
  • The upstream harness runs filtered cases only, either opt-in, periodic, or as a small curated CI job.
  • We do not depend on the moving/full upstream suite for merge correctness.
  • Any upstream failure we decide matters becomes an original rshell scenario test.

That gives us the benefit of gawk coverage without the full cost/flakiness/control problem

@matt-dz matt-dz changed the title test(awk): add external upstream harness scaffold test(awk): add rewritten scenario harness scaffold May 7, 2026
@matt-dz matt-dz force-pushed the codex/awk-bwk-tests branch 2 times, most recently from cf898c3 to d8e1cb4 Compare May 8, 2026 13:08
@matt-dz matt-dz changed the title test(awk): add rewritten scenario harness scaffold test(awk): add rewritten oracle harness May 11, 2026
@matt-dz matt-dz force-pushed the codex/awk-bwk-tests branch from 1e3539e to 985be3d Compare May 11, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants