configure warnings as errors in the test suite

[graingert]

Feature or enhancement

Proposal:

It's very easy (especially when deprecating something) to introduce a lot of noisy test output with new warnings, eg: #127949 (comment)

I think it would be useful to either configure warnings as errors in the test suite (and then capture unraisable/threadexcept errors using a mechanism like pytest has) or tee any warnings produced, and fail CI if any have been produced that were not captured with catch_warnings or assertWarns etc like https://pypi.org/project/pytest-max-warnings/

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/its-too-easy-to-introduce-noise-in-cpythons-test-suite-output-resourcewarnings-and-deprecationwarnings/75234

Linked PRs

• gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings #128726
• gh-128770: fix ResourceWarning in test_pyrepl #128906
• [3.13] gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings (GH-128726) #128935
• Revert "gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings (#128726)" #128936
• gh-128770: raise warnings as errors in test suite - except for test_socket which still logs warnings, and internal test warnings that are now logged #128973
• [3.13] gh-128770: fix ResourceWarning in test_pyrepl (GH-128906) #129179
• [3.13] gh-128770: raise warnings as errors in test suite - except for test_s… #131802
• [3.12] gh-128770: raise warnings as errors in test suite - except for test_s… #131824

[sobolevn]

See #110012 test_socket raises warnings right now.

[graingert]

See #128816 test_doctest also has a warning

[graingert]

See #110012 test_socket raises warnings right now.

I'm thinking we mark these as default with catch_warnings in the test for now?