chore: deprecate pylint in favor of ruff

[mistercrunch]

This could be slightly controversial, but submitted as a DRAFT for discussion. After thinking about it, I think we should just move forward with this. Shouldn't be controversial. File under simplify/accelerate builds/devex/tooling.

The case for pylint deprecation:

• mostly covered by ruff: what matters most in pylint is already re-implemented
• pylint is SLOW, ruff is a million times faster - pre-commit / pylint is one of our longest, most expensive CI step
• rules are currently scattered in different places and can conflict at times
• checked in with the Airflow community, they went all in on ruff, deprecated pylint and are extremely happy with the switch

Very relevant is this link that explains what pylint features are covered or not by ruff:
astral-sh/ruff#970

Essentially we can simplify & speed things up here..

If we agree on the direction, the plan is to follow up and enable all the rules, apply ruff check --fix and ruff check --add-no-qa, which will enable the set of rules commented now, fix the obvious, and add a # noqa to the lines that are not immediately fixable.

[michael-s-molina]

@mistercrunch Can we reserve this for the 5.0 breaking window? We'll gather proposals for 5.0 next week. I'll send an email and you could include this. I know that this will break our internal release so using a major release is safer.

[mistercrunch]

Can we reserve this for the 5.0 breaking window

When is this scheduled? Thinking about it though, switching dev tools shouldn't have any impacts on UX/interfaces. From my understanding semver is mostly interface oriented. I'm favoring moving fast on this while I have momentum on improving devex. Though it may have impact on mergeability of cherries, so may be good to wait for that.

Also noting that when I enabling the pylint rules currently commented in my PR using ruff, it finds hundreds lint issues, meaning turning on ruff should up code quality moving forward (my plan is to enable these rules, run --fix, and --add-noqa).

[mistercrunch]

@sadpandajoe curious on your thoughts about timing, for context the follow up PR would touch the backend with minor linting tweaks, maybe 300 LoC across the backend.

[sadpandajoe]

@sadpandajoe curious on your thoughts about timing, for context the follow up PR would touch the backend with minor linting tweaks, maybe 300 LoC across the backend.

@mistercrunch I believe the breaking change window is opening soon. I don't think it'll affect Preset but sounds like it might have an affect on AirBnb?

[villebro]

I was hesitant at first, but given all the persistent unfixed critical issues on pylint, and the momentum ruff has, I agree this is the future right now. LGTM with a minor typo nit.

[mistercrunch]

@michael-s-molina do you still think we should hold back on the 5.0 process or can I just merge?

[villebro]

Hmm I don't think this should necessarily be tied to the 5.0 breaking window 🤔 I say 🚢 it, but let's wait for @michael-s-molina to confirm.

[michael-s-molina]

I'm ok with merging this now if you think it's not controversial. I would just ask to add something to UPDATING.md as there are orgs that extend Superset's code and are currently using pylint. I'll remove this PR from the 5.0 board.

[mistercrunch]

Ok adding a note in UPDATING.md and mergin'! 🚢

[mistercrunch]

Ok, will merge and fast follow with another PR enabling more ruff rules (uncommenting what's here https://github.com/apache/superset/pull/31262/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R290-R302) that in some edge cases collided between ruff --fix and pylint (they disagreed on technicalities around enforcing/fixing certain rules - both takes were sane but not exactly the same in all cases)

[mistercrunch]

Fast-follow here -> #31447