test_sqlite3 fails on ZFS

[larryhastings]

Bug report

Bug description:

If I build CPython 3.13.5 on my Linux workstation in my home directory, test_sqlite3 fails. If I build CPython 3.13.5 in /tmp, test_sqlite3 works fine. My /home partition uses ZFS, which is somewhat rare, so I'm betting that's why it fails for me.

I ran the test using

% ./python -m test -v test_sqlite3

Here's the output of that program when run in my home directory.

test_sqlite3.failure.output.txt

Current CPython trunk also fails in the same way, except now the output is colorized.

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-136289: Fix test_sqlite3 on platforms with strict UTF-8 filesystem #136326
• [3.14] gh-136289: Fix test_sqlite3 on platforms with strict UTF-8 filesystem (GH-136326) #136350
• [3.13] gh-136289: Fix test_sqlite3 on platforms with strict UTF-8 filesystem (GH-136326) #136351

[brianschubert]

I'm getting an error page for the attached output file. Could you try re-uploading or adding the output inline in a <details> block?

[larryhastings]

Sure, here's just the errors part of the log.

======================================================================
ERROR: test_open_undecodable_uri (test.test_sqlite3.test_dbapi.OpenTests.test_open_undecodable_uri)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/larry/src/python/buildtrunk/Lib/test/test_sqlite3/test_dbapi.py", line 699, in test_open_undecodable_uri
    with contextlib.closing(sqlite.connect(uri, uri=True)) as cx:
                            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
sqlite3.OperationalError: unable to open database file

======================================================================
ERROR: test_open_undecodable_uri (test.test_sqlite3.test_dbapi.OpenTests.test_open_undecodable_uri)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/larry/src/python/buildtrunk/Lib/test/support/os_helper.py", line 373, in unlink
    _unlink(filename)
    ~~~~~~~^^^^^^^^^^
OSError: [Errno 84] Invalid or incomplete multibyte or wide character: b'@test_398849_tmp\xe7w\xf0'

======================================================================
ERROR: test_open_with_undecodable_path (test.test_sqlite3.test_dbapi.OpenTests.test_open_with_undecodable_path)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/larry/src/python/buildtrunk/Lib/test/test_sqlite3/test_dbapi.py", line 652, in test_open_with_undecodable_path
    with contextlib.closing(sqlite.connect(path)) as cx:
                            ~~~~~~~~~~~~~~^^^^^^
sqlite3.OperationalError: unable to open database file

======================================================================
ERROR: test_open_with_undecodable_path (test.test_sqlite3.test_dbapi.OpenTests.test_open_with_undecodable_path)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/larry/src/python/buildtrunk/Lib/test/support/os_helper.py", line 373, in unlink
    _unlink(filename)
    ~~~~~~~^^^^^^^^^^
OSError: [Errno 84] Invalid or incomplete multibyte or wide character: b'@test_398849_tmp\xe7w\xf0'

----------------------------------------------------------------------

[picnixz]

It appears that undecodable paths are not supported on ZFS. Could it be your current locale or your fs config that only accepts ASCII chars?

It might also be on our side where we should skip tests on filesystems that don't support wide characters in filenames (it appears that we can create the file but we can't delete it for some reason so it could also be an issue with unlink itself or the way we cleanup resources).

I may suspect that there is maybe a PyUnicode_DecodeFS() call that doesn't work. What are the other tests that are failing? are they failing for similar reasons?

[oskar-skog]

Related: #81765

I don't think this is a locale issue, it appears that ZFS can be configured to not allow invalid UTF-8 in filenames.

[emmatyping]

Yeah, zfs has normalization and utf8only properties (https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#normalization), which limit the filenames to only valid utf8 sequences. I am a bit surprised that the test suite would be able to create a file in this mode that it couldn't read.

[StanFromIreland]

We have had issues like this before (#133677).

I am unable to reproduce even with my filesystem encoding as ascii.

I am a bit surprised that the test suite would be able to create a file in this mode that it couldn't read.

From this comment it appears it is possible, on Windows at least.

cpython/Lib/test/support/os_helper.py

Lines 125 to 129 in 5dac137

	# b'\xff' is not decodable by os.fsdecode() with code page 932. Windows
	# accepts it to create a file or a directory, or don't accept to enter to
	# such directory (when the bytes name is used). So test b'\xe7' first:
	# it is not decodable from cp932.
	b'\xe7w\xf0',

cc @serhiy-storchaka

[serhiy-storchaka]

Please test #136326.