Testing third-party stubs in isolated environments

[srittau]

At the moment, whenever there is change to typeshed, we test all stdlib and third-party packages in the same testing environment, using the --custom-typeshed-dir so that all packages can see each other. There are several practical problems with that:

• We can't test the requires fields for correctness; all packages are always available.
• We can't have packages that depend on external packages, see for example Allow non-types dependencies #5768, Remove cryptography (depends on #5952) #5618, paramiko: Change dependencies on types-cryptography, allow using upstream type hints #5847, and the discussion in Allow non-types dependencies #5769 (all related to cryptography). (Except if we'd just install all non-types dependencies for all package into our venv.)
• It doesn't scale if we increase the number of third-party packages in typeshed.
• Ideas like per-package configuration is not possible at the moment, see for example mypy tests: Allow per distribution strictness options #1526.

My suggestion: Only run the tests for the third-party stubs that have actually changed, and run each test in a separate environment. This environment only has the requirements from METADATA.toml installed. This fixes the problems above and also means:

• The stubs will not be able to use features from stdlib that are not yet in the latest type checker distributions.
• Changes that depend on changes in another third-party stub need to wait until that change has been released. (Happens every three hours, so it shouldn't be too much of a problem.)
• Running the tests for all stubs will be slower, since each package needs its own venv installation.

[Akuli]

A possible downside: If the tests of a third-party stub break for some reason, e.g. because a new version of the corresponding non-stub package is released, the problem will remain unnoticed until someone makes a PR for that specific stub package. I remember most pull requests showing red CI because of some Pillow thing, but can't find it from the PR history now.

[srittau]

It would be useful to run full tests at a regular interval, say once per day or week.

[hauntsaninja]

Note the stubtest third party code already does this venv creation. We could steal that code / maybe use some fancy Github Actions CI caching to cache venvs across workflows. Switching to testing only changed third party stubs is probably a better idea than just doing everything and sharding, which is the approach test_stubtest currently takes.

Like you say, I'd want to think through how we manage different stdlib and types-* versions; to that goal, here are some previous issues we've had on those lines: #4815 #5786 #5751

[Akuli]

This seems to be done now.

[srittau]

No, we still need to use separate venvs for each distribution, so that the dependencies don't interfere with each other.

[hmc-cs-mdrissi]

I would be interested in working on this mainly to make it possible to have non-types dependencies.

Would reasonable steps in order be,

1. Adjust each check to only run for changed folders. Start with mypy then continue to pyright then pytype.
2. Create a new venv for each folder checked. Again mypy -> pyright -> pytype
3. Add support for reading metadata.toml and install dependencies for given folder.
4. Add support for per package mypy.ini/pyrightconfig.json/etc files.

Any major tasks I'm missing?