RTC: Fuzz testing

[dmsnell]

Fuzzing the Realtime-Collaboration (RTC) feature in the editor

This tracking ticket is meant to provide an overview of detecting and proposing fixes for concurrency bugs in the RTC subsystem. This work should be part of the WordPress 7.0 release (including backports into Core), consisting of bugs/defects in the proposed beta/release-candidate code.

As this work is primarily driven by LLMs, many of the reports may be suspect until demonstrated to be real.

Findings

Tracking in Core-65138

• RTC: potential data corruption on block merge write path due to rich-text text offset directly into a diff API that operates on HTML string indices #77532
  • RTC: fix #77532, rich text offset mismatch with diff API #77658
• RTC: add regression tests for data corruption bug due to cursor scope issue #77662
• RTC: Fix title divergence between users on page refresh after title update #77666
• RTC: fix connection lost error on large update cause by mismatch between update size bounds check and expanded base64 update size #77669
• RTC: fix cursor awareness / presence bug in nested rich text elements #77673
• RTC: Fix race condition on room creation which can cause a split update log #77675
  • RTC: Ensure a single canonical update log for collaborative edits. wordpress-develop#11660
• RTC: clients can send malformed input that causes other clients' editors to crash #77678
• RTC: Fix issue where undo acts on the wrong synced entity #77681
• RTC: Fix merge bug where client applies stale local block snapshot after receiving remote CRDT update #77775
• RTC: fix interaction with autosave that can cause posts and revisions to get lost even when a single user is editing a post #77865
  • RTC: Backport fix to prevent autosaves from losing content. wordpress-develop#11691
• RTC: fix table edit loss and associated revision loss when an old or externally-created post has no persisted CRDT document and contains duplicate table rows #77866
• RTC: fix stale block snapshot overwriting newer state #77876
• The server allows overwriting with a stale document
  • RTC: Add base version tracking to prevent stale CRDT doc overwrites #77889
  • RTC: Reject stale persisted CRDT documents #77890

Unrelated to RTC

• Tests: Try reviewing a specific E2E test. #77632
• Fix wait / sleep which is causing many PRs to fail performance tests in CI for reasons unrelated to the PR #77726

[danluu]

Noting a couple issues in the AI generated fixes. In #77681, the original test and fix had significant gaps. They did fix an issue, but in an incomplete way. I very strongly suspect further fuzzing would have found that issue and closed the gaps, but the initial PR wasn't very good. I would rate this below the quality I'd expect from a reasonable human, but the process here would have eventually produced a good result, just with more churn than a reasonable human would've put into the system.

#77676 was a duplicate. I'm not (attempting to) do AI generated GitHub interactions and did do any kind of AI search for dups (I used normal GitHub search and searched for different keywords), so that one is directly on me.

In terms of workflow, at one point, I tried using Conductor (conductor.build), which has been recommended by multiple people I know at other companies. This had the following issues:

• It generated PRs even though it wasn't supposed to (there's an AI process for checking that issues are valid before surfacing them to me as valid issues and it bypassed that process). These PRs were also quite broken. For example, see the busted description in [Accidental PR creation, please ignore] #77674
• Issues in different Conductor Workspaces would contaminate each other for some reason. A request to fix the CI failure in one issue would cause pushes to a different PR. Things like this happened multiple times with only a small amount of use. I confirmed that this was not a problem where I typed into the wrong window by looking at the history. Conductor Workspace isolation did not work as intended here, and issue I never had with my previous workflow
• Conductor doesn't seem to be intended to be used with what one might call "higher order" workspaces. It's easy to create workspaces "by hand" and work in each one and this can be automated in a janky way, but what I really want is to be able to have me or an agent pass it a list of issues, each one of which has some actions done to it (basically, like a map or a loop), but it's really not made for that

On the bright side, we have no known false positives (yet) that weren't automatically rejected by AI review after 4 issues were reviewed by a human who's familiar with Gutenberg (I'm not familiar with Gutenberg myself and could easily think a false positive were real if I was the only one looking at things). I'm sure our false positive rate isn't 0, so we'll get some at some point, but this fuzzer seems to be working ok at the moment and seems like a net positive even if we get a moderate number of false positives going forward.

[danluu]

Here's another code issue, which seems like a classic AI generated code thing: In #77726, codex apparently duplicated a couple of functions in tests: #77726 (review).

[danluu]

Noting an issue that hasn't caused a known problem in a filed a PR or issue yet, but is where a lot of time for this project is going. For completely reasonable reasons, this project has an existing work flow and only merges PRs from random outsiders like me at some rate.

The rate this fuzzer can find bugs at is much higher, so there's an increasing backlog of bugs. The problem is that, as the backlog increases, fuzzing effectiveness decreases. Fuzzing against trunk almost exclusively returns known failures. Even if these are throw away by the triage process, that uses quite a few tokens and it's fairly certain that real failures are being masked by these other failures because tests that would've failed on a new failure fail on a known failure first.

The obvious fix for this is to fuzz against a branch with all known fixes, but this also becomes less effective as the number of fixes increases. One of the big issues there is that, as the number of outstanding PRs increases, the changes that will happen when the PRs get merged also increases, so the fuzzing is running against an increasingly divergent codebase relative to what trunk will be after the PRs get merged. #77887 (stale local snapshots overwriting object+query map operations) is an example of the kind of PR that causes this issue. It's a fairly invasive change with multiple different approaches to solving the problem. If the approach that ends up getting committed is different from the proposed approach, fuzzing will end up running against fairly different code.

What are possible solutions to this problem?

1. Merge PRs with much less scrutiny
2. Find other things to fuzz that are more independent
3. Do something completely different while the queue drains
4. ???

(1) would probably increase quality, in the sense that the number of bugs users are hitting would go down. However, it would probably decrease quality, in the sense that, whatever abstract notions of quality exist about the code base are (such as code quality), these will probably not be followed as well by an LLM today as by humans even if the LLM is given guidance. #77675 (race condition at room creation causes split rooms with respect to updates) is an example of this. From a correctness standpoint, it's very clear that's a serious problem, both in terms of severity when the bug is observed and the likelihood of the bug occurring. However, there's a notion of what should and should not be done there that, in the general case, would be difficult to write down in advance. With feedback on that PR, it could possibly be pushed into the shape desired by maintainers, but extracting a list of all such rules in advance is beyond the capabilities of an LLM today and I'm not sure we could get maintainers to write down the complete list in advance either.

(2) has a number of possibilities. Of course there are completely different projects. Additionally, other parts of Gutenberg could be good targets. Subjectively, I feel like I run into a bug every time I use the block editor. I doubt that's literally true, but it's fairly common for me to hit 3-4 bugs in the course of a single use of the block editor. I'd expect that most of those bugs are fairly independent from these RTC fixes and it might take an hour fuzzing to hit the point we're at here (which we hit within an hour here, although I've been pushing through that with an increasingly inefficient workflow since this is somewhat widely deployed today and there's a plan to deploy this much more widely in the near future).

(3) Not much to say about this.

(4) Any other ideas?