support pep 673 Self type

[zzzeek]

Feature

Support the pep-673 Self type

pep is at: https://www.python.org/dev/peps/pep-0673/

a rough idea of it is that this code:

from typing import Self

class Widget:
    def add_thing(self, value: str) -> Self:
        self.value = value
        return self

is quasi-equivalent to:

from typing import TypeVar

TWidget = TypeVar("TWidget", bound="Widget")

class Widget:
    def add_thing(self: TWidget, value: str) -> TWidget:
        self.value = value
        return self

Pitch

the appeal of the pep is that for the very common pattern of method-chained object construction, methods can easily indicate they return an object of the same type upon which the method is being invoked. For example if I made class SubWidget(Widget), the SubWidget.add_thing() method reports SubWidget as the return type automatically. The pattern using TypeVar seems to be more or less equivlant, but is more verbose requiring the declaration of TypeVar objects per class hierarchy as well as that it has to be explicitly present on the "self" parameter in methods.

We are looking for this feature to make our job of integrating typing into SQLAlchemy an easier job. it seems to be a fairly straightforward translation between two idioms.

cc @CaselIT

[erictraut]

FWIW, both pyright and pyre have implemented this feature in its currently-documented form if you want to play with it.

[zzzeek]

FWIW, both pyright and pyre have implemented this feature in its currently-documented form if you want to play with it.

I know :) it's great. I hate when people email SQLAlchemy with a feature request and say " django does it! " :) but yeah, just trying to see if people using mypy can take advantage also in the near term or if there's no roadmap for this one here.

[AlexWaygood]

There is already a draft PR working on adding support for PEP 673: #11666

[zzzeek]

@AlexWaygood we are literally trying to choose right now if SQLAlchemy 2.0, for which we hope to have beta releases by the end of Q1 2022, can use pep673 Self or not. I know this is normally not something projects can estimate but is it your experience a PR like the one suggested in #11666 can move towards being merged within a few months or is it likely it could stall for a long time?

[zzzeek]

at the same time, if we do release 2.0 using pep 673, mypy users would be more interested in seeing this PR completed so for that reason alone I might decide to use it now.

[AlexWaygood]

Hey @zzzeek -- unfortunately, I'm not really the right person to ask about this :// I'm not actually on the mypy core team, and have only contributed a few documentation fixes to mypy -- I tend to contribute more towards the CPython & typeshed repos, I just stalk the mypy issue tracker from time to time 😄 Perhaps @hauntsaninja might be able to shed some more light?

Having said that, I think the approach typeshed has taken would probably serve you well! You could define a single TypeVar in a standalone module like this (let's call it typing_utils):

# typing_utils.py
from typing import TypeVar

Self = TypeVar('Self')

Then, in the rest of your code base, you could use that TypeVar like this:

# foo.py
from typing_utils import Self

class Foo:
    def returns_self(self: Self) -> Self:
        return self
    @classmethod
    def returns_cls(cls: type[Self]) -> type[Self]:
        return cls

In that way, you'd get type annotations that work perfectly for now; and if PEP 673 is accepted and implemented, you could quite easily run a single script using AST to clean up your code base, converting Foo into the following:

# foo.py
from typing_extensions import Self

class Foo:
    def returns_self(self) -> Self:
        return self
    @classmethod
    def returns_cls(cls) -> type[Self]:
        return cls

The old version of Foo (using pre-PEP 673 syntax) and the new version (using PEP 673 syntax) would have semantically identical type annotations from mypy's perspective.

[hauntsaninja]

I also just stalk the mypy issue tracker from time to time :-) I don't think there is anyone who gets paid time to work on mypy, so there isn't a concrete roadmap. In my experience, you can expect a lag time of a couple months from contributor-completes-PR to mypy-release-that-contains-said-PR.

Re PEP 673 specifically:
I haven't looked at the linked PR, but in the abstract PEP 673 doesn't seem like a hard thing to implement.
I'll also note that PEP 673 has not yet been officially accepted, so I'd keep that in mind when thinking about what promises you're comfortable making to SQLAlchemy users. AlexWaygood's suggestion of just using a normal type variable (and ignoring the bound) + codemodding at a later time is a good option to consider.

[zzzeek]

Having said that, I think the approach typeshed has taken would probably serve you well! You could define a single TypeVar in a standalone module like this (let's call it typing_utils):

# typing_utils.py
from typing import TypeVar

Self = TypeVar('Self')

OK, we can certainly go with that, the approach we have at the moment is copied from how pep-673 illustrates it, with individual TypeVars that are each "bound" to the target type. There should be no reason that's needed though, right? that is, the "Self" that's not bound will work just as well?

[AlexWaygood]

For situations like this, the bound argument is, unfortunately, still necessary, yes — in an ideal world, mypy should be able to understand that TypeVars in that situation are implicitly bound, but that hasn't been implemented yet.

It depends whether you're using mypy to check function implementations, or just writing stubs, though. If you're just writing stubs, you can get away with a lot more, and the bound argument is usually not required 🙂

[zzzeek]

this is inline typing in SQLAlchemy for 2.0 where we will no longer use stubs.

[AlexWaygood]

this is inline typing in SQLAlchemy for 2.0 where we will no longer use stubs.

In which case, yeah, you'll often need the bound argument, unfortunately, or you'll find mypy will erroneously complain about things you're doing inside your functions 😕 but, it still may be worth adding some kind of 'tag' to these TypeVar names (such as having the "Self" suffix with all of them) so that you can easily identify them in the future and change them quickly if/when PEP 673 is accepted.

from typing import TypeVar

FooSelf = TypeVar('F', bound='Foo')

class Foo:
    bar: int
    
    def change_bar(self: FooSelf, x: int) -> FooSelf:
        self.bar = x  # mypy erroneously complains about this line if the TypeVar is not bound 
        return self

[zzzeek]

yes that's what we've been doing, thanks for the tips!

[AlexWaygood]

yes that's what we've been doing, thanks for the tips!

No problem!