Skip to content

Disallow crawling 3.9 docs#332

Merged
StanFromIreland merged 4 commits intopython:mainfrom
StanFromIreland:3.9-robots
Apr 1, 2026
Merged

Disallow crawling 3.9 docs#332
StanFromIreland merged 4 commits intopython:mainfrom
StanFromIreland:3.9-robots

Conversation

@StanFromIreland
Copy link
Copy Markdown
Member

No description provided.

@hugovk
Copy link
Copy Markdown
Member

hugovk commented Apr 1, 2026

Well spotted.

I was going to suggest adding this to the checklist at https://peps.python.org/pep-0101/#moving-to-end-of-life so we don't forget next time, but in the interest of saving me work in the 2030s, can we generate it from https://peps.python.org/api/release-cycle.json ?

@StanFromIreland
Copy link
Copy Markdown
Member Author

in the 2030s

I tried to mock that in the test ;-)

Disallow: /3.9/
{% for version in versions -%}
{% if version.status == "EOL" -%}
Disallow: /{{ version.name }}/
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this include all of /2/ and /2.0/ - /3.10/?

Will it add any others?

https://peps.python.org/api/release-cycle.json only goes back to 2.6.

https://peps.python.org/api/python-releases.json has more but is more plumbing. We could hardcode the oldest ones, and the newer ones will be updated automatically.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted to hardcode the old ones, with that, it adds one line:

+ Disallow: /3.9/

Copy link
Copy Markdown
Member

@hugovk hugovk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@StanFromIreland StanFromIreland merged commit 2323166 into python:main Apr 1, 2026
13 checks passed
@StanFromIreland StanFromIreland deleted the 3.9-robots branch April 1, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants