Testsuite

[thomie]

Hello!

I have created a test-suite for evil-helix and possibly other vim-alike editors.

You can check it out here: https://github.com/thomie/vim-keymap-test

Each test defines the contents of the input buffer (for example foo bar), a list of commands to run (for example [dw]), and the expected output (bar in this case).

For each test, the test runner starts vim or (evil-)hx as an external process, sets the input buffer, simulates a user executing the commands, and compares the output with the expected output.

I have found it quite useful while hacking on evil-helix (merge requests incoming!), e.g. to prevent regressions.

Let me know what you think.

[usagi-flow]

You know, yesterday I was just thinking about tests, and how I‘m missing something like blackbox tests; tests that don‘t need to know any implementation detail, and that could be run against both Vim and evil-helix. Tests that can also remain untouched, should upstream Helix change too considerably (e.g. for the scheme based configuration).

It seems like you implemented pretty much that idea.

That might actually be a good candidate for running in GH Actions workflows.

[thomie]

Ah yes, blackbox testing is what it's called. I updated the description.

So let's collaborate.

Before trying it in GH Actions, it would be nice to fix the 2 second delay that currently happens with every test. See the limitations section of the README.

Or perhaps use another implementation entirely?

I just picked expect because it showed up in google searches as the tool to use for testing terminal programs. I also briefly tried empty, and liked that it's a very small program (expect is huge), but I couldn't get it to work with Vim (not sure if it's even possible). I also looked at Pexpect, which is just expect but written in Python.

I had some experience with shelltestrunner already.

All I'm saying is, I'm not attached to any of this. Let's use what works best.

[usagi-flow]

Thank you for sharing your insights on that.

While I also had no particularly strong feeling toward any option, you did get me intrigued by Pexpect a bit more. That's mostly because it's using a general purpose language (and a simple one, which should make it possible even for a non-developer to write tests).

I quickly tried it out and wrote a very simple test that switches to visual and back, then quits, and I like it. Pexpect behaves as I'd expect it to (no pun indended).

#!/usr/bin/env -S uv run

import pexpect

print("Start evil-helix")
child = pexpect.spawnu("hx", timeout=3)

print("Expect an initial line number")
child.expect("    1")

print("Press v")
child.send("v")

print('Expect " VIS " in the mode indicator')
child.expect(" VIS ")

print("Press escape")
child.send("\x1b")

print('Expect " NOR " in the mode indicator')
child.expect(" NOR ")

print("Quit")
child.sendline(":quit")

Interestingly, I also have that 2s delay at the beginning. It's weird; maybe it is coming from the editor after all. But that shouldn't be a blocker for integrating such tests in the CI.

I'll give these thoughts another day or two, but these tests, written with Pexpect, might be a very interesting way to go.

If we move forward with this, I'm considering having these blackbox tests located in a separate repository in the github.com/evil-helix organization (where I'm planning on moving this repository to as well in the future). We could then write the GH Actions workflows that invoke the tests of that other repository.

[thomie]

Yes Pexpect together with perhaps a test framework/library is a great option too, and probably easier to understand.

If we move forward with this, I'm considering having these blackbox tests located in a separate repository in the github.com/evil-helix organization (where I'm planning on moving this repository to as well in the future). We could then write the GH Actions workflows that invoke the tests of that other repository.

Sound good.

[usagi-flow]

Small update on this issue, because I've been digging into this topic every now and then.

I realized that Pexpect and similar frameworks seem to be made for simple input/output CLI applications, and less for TUIs.
More specifically, Pexpect seems to lack a good implementation of a virtual screen, which, to be honest, isn't trivial. VTE codes must be implemented, and a state must be managed.

What Pexpect does, is analyzing the output stream as a stream of characters. That's a problem when TUI applications (like evil-helix) want to modify something (e.g. the editor mode) somewhere on the screen (bottom left). They don't redraw the whole screen, but instead "jump" to the bottom left with a special escape sequence, and then proceed from there. This makes it very hard to just "check the current mode" - which would be a very important check in blackbox tests.

I tried to combine Pexpect with Pyte, but i had to manually forward the Pexpect stream of characters to the Pyte screen, which I didn't like.

I also had a look at the rust equivalents, rexpect and memterm, but I felt like both projects were at earlier stages than Pexpect and Pyte respectively.

Long story short: I'm open to ideas.