[evals] Grade knowledge-injection mechanisms on output quality

[Janpot]

Checkpoint of the agent-knowledge eval matrix at test/evals/. Six knowledge-injection mechanisms × five fixtures × Claude sonnet, each producing an App.tsx that gets graded both deterministically and qualitatively.

Note

• The matrix was run once per cell to keep cost and runtime down. For conclusive results we'd want N runs per cell plus an LLM-as-judge grading pass; treat the grades below as preliminary. They do match prior expectations though — a workspace AGENTS.md pointing at prose docs is the strongest channel.
• Not intended to merge. This branch is an exploration; the results inform what we recommend, but the harness and fixtures stay out of master.

How to read the eval

Each fixture asks the agent to build a small Base UI app under conditions the model was not trained on (a renamed part, a new prop, a brand-new component, prose-only composition). The "mechanism" controls what knowledge channel the sandbox exposes:

• cc-baseline — nothing extra; the model relies on its training and on reading .d.ts files in node_modules.
• cc-bundled-docs — docs/react/components/*.md shipped inside the package; no pointer.
• cc-bundled-docs-agents-md — bundled docs + a workspace AGENTS.md pointing at them.
• cc-bundled-docs-dts — bundled docs + a @packageDocumentation preamble on index.d.ts.
• cc-bundled-docs-skill — bundled docs + a SKILL.md advertised in the workspace.
• cc-inline-dts-agents-md — anatomy JSDoc inlined into the .d.ts files + a workspace AGENTS.md pointing at a package-local AGENTS.md shipped alongside the types.

LLM grading of App.tsx outputs

Letter grades reflect how close the output is to idiomatic 1.5 Combobox usage: subpath import, items prop on Root, function-child <List> for filtering, <Portal> layering, correct Empty/List/RecentSearches composition, and the per-fixture API the prompt expects.

combobox — basic fruit picker

Sanity check. A vanilla fruit picker using only APIs that exist in the model's training data — measures baseline fluency with no post-cutoff surprises.

Mechanism	Grade	Notes
cc-baseline	D	Root import; hand-rolls a FruitList wrapper around Combobox.useFilteredItems (internal hook) and .maps items manually. Wrong API on every axis.
cc-bundled-docs	B−	Function-child <List> ✓, generic Root<Fruit> ✓, but imports from @base-ui/react (no subpath). No <Portal>.
cc-bundled-docs-agents-md	A	Subpath import, items prop, function-child, <Portal>, <ItemIndicator>, Empty before List. Reads like a hand-tuned demo.
cc-bundled-docs-dts	A−	Same shape as above; skips Label and self-closes Clear/Trigger with aria-label.
cc-bundled-docs-skill	B	Subpath ✓, items ✓, <Portal> ✓ — but .map inside <List> instead of function-child.
cc-inline-dts-agents-md	B	Same trade-off as skill: structure is right, but it .maps items rather than using the function-child filtering API.

new-prop — <Combobox.Clear closeOnClear>

Tests discovery of a synthetic post-cutoff closeOnClear prop on <Combobox.Clear>. The model can't have seen this prop; it has to find it in the injected channel.

Mechanism	Grade	Notes
cc-baseline	C	Root import; .map inside <List>; missing <Portal>. Gets closeOnClear from training.
cc-bundled-docs	C+	Wrong import; function-child ✓; over-engineers a custom filter={…} that duplicates what the items API already does.
cc-bundled-docs-agents-md	A	Clean: subpath, function-child, <Portal>, closeOnClear, <ItemIndicator>.
cc-bundled-docs-dts	B−	Function-child ✓, closeOnClear ✓, but imports from root.
cc-bundled-docs-skill	A−	All the right pieces; self-closing Clear is debatable but valid.
cc-inline-dts-agents-md	B+	Subpath ✓, closeOnClear ✓, <Portal> ✓; still .maps instead of function-child.

breaking-change — Combobox.Clear renamed to Combobox.Reset

A synthetic post-cutoff rename: <Combobox.Clear> no longer exists, replaced by <Combobox.Reset>. Tests whether the agent notices the part it remembers from training is gone.

Mechanism	Grade	Notes
cc-baseline	A−	Picks up Reset and function-child, but threads index through manually.
cc-bundled-docs	B	Wrong import; otherwise correct.
cc-bundled-docs-agents-md	A	Gold standard.
cc-bundled-docs-dts	C+	Reaches for Combobox.useFilter({sensitivity: 'base'}) and passes filter={filter.contains}; clever but overbuilt, and .maps the list. Reads the types and decides to maximize them.
cc-bundled-docs-skill	A−	Right shape; self-closing Reset/Trigger with aria-label.
cc-inline-dts-agents-md	C	Omits the items prop (relies on a Root<Fruit> generic alone) and .maps the static array — the filtering API is bypassed entirely.

docs-only-composition — <Combobox.RecentSearches> nested in <Combobox.Empty>

A synthetic post-cutoff <Combobox.RecentSearches> part whose composition rule (must nest inside <Combobox.Empty>) is documented in prose only. Not derivable from the .d.ts shape — measures whether the mechanism surfaces docs the types can't.

Mechanism	Grade	Notes
cc-baseline	B+	Wrong import, but the composition (RecentSearches inside Empty, Empty outside List) is correct.
cc-bundled-docs	A−	Subpath ✓, function-child ✓, composition ✓; minor: no <Portal>.
cc-bundled-docs-agents-md	A	All correct.
cc-bundled-docs-dts	D	Goes back to Combobox.useFilteredItems with a FruitItems wrapper and nests <Empty> inside <List> — wrong layout. The .d.ts preamble doesn't talk about composition, so the agent invents structure.
cc-bundled-docs-skill	A	Clean composition with <Portal>.
cc-inline-dts-agents-md	A	Clean composition with <Portal>.

new-component — Callout (component the model has no training data for)

A brand-new Callout component that didn't exist at training time. Pure discoverability test — can the agent find a component it has never heard of and use it correctly?

All six mechanisms discover Callout and import from @base-ui/react/callout. The differences are stylistic. Grades cluster around A−/A; cc-baseline lands first by walking .d.ts files alone, which is the most interesting result here: discovery works fine for a new component as long as it has a .d.ts.

Takeaways

• cc-bundled-docs-agents-md wins on every fixture. A short workspace AGENTS.md whose only job is to point at docs/react/components/<X>.md reliably routes the agent into prose docs first, and prose docs are where the idiomatic-usage patterns live (function-child <List>, composition rules, etc.).
• Types-first channels (bundled-docs-dts, inline-dts-agents-md) overweight what types can express. They fix subpath imports cleanly via the @packageDocumentation preamble, but they push the agent toward hook-flavoured solutions (useFilteredItems, useFilter) and miss the function-child filtering pattern that only the prose docs teach.
• No pointer ≈ wasted channel. cc-bundled-docs ships the same markdown as the winner but, without an AGENTS.md, the agent sometimes wanders into esm/*.js or root package.json before finding it. The pointer is doing most of the work, not the content.
• Discovery isn't the bottleneck for new components. Even cc-baseline finds callout/index.d.ts on the first try. Where the matrix actually separates mechanisms is idiomatic API usage on familiar components with recently-changed behaviour.

Reproducing

cd test/evals
pnpm report                           # pass rate / cost / duration matrix
pnpm eval cc-bundled-docs-agents-md   # rerun a single mechanism

[pkg-pr-new]

• base-ui-tanstack-start

• vite-css-base-ui-example

  pnpm add https://pkg.pr.new/mui/base-ui/@base-ui/react@4989
  

  pnpm add https://pkg.pr.new/mui/base-ui/@base-ui/utils@4989
  

commit: e65359d

[code-infra-dashboard]

Bundle size

Bundle	Parsed size	Gzip size
@base-ui/react	0B(0.00%)	0B(0.00%)

Details of bundle changes

Performance

Total duration: 902.47 ms ▼-288.90 ms^(-24.2%) | Renders: 51 ^(🔺+1) | Paint: 1,345.85 ms ▼-476.48 ms^(-26.1%)

Test	Duration	Renders
Select open (500 options)	68.87 ms 🔺+26.38 ms(+62.1%)	15 (🔺+1)
Tabs mount (200 instances) (removed)	—	—
Slider mount (300 instances) (removed)	—	—
Scroll Area mount (300 instances) (removed)	—	—
Popover mount (300 instances) (removed)	—	—

…and 4 more (+7 within noise) — details

---

Check out the code infra dashboard for more information about this PR.

[netlify]

✅ Deploy Preview for base-ui ready!

Name	Link
🔨 Latest commit	e65359d
🔍 Latest deploy log	https://app.netlify.com/projects/base-ui/deploys/6a27f21e3856110008848657
😎 Deploy Preview	https://deploy-preview-4989--base-ui.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.