Skip to content

Replace asyncio and httpx with urllib3 and ThreadPoolExecutor #357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hugovk merged 2 commits into from
Sep 16, 2026
+46 −43
Merged

Replace asyncio and httpx with urllib3 and ThreadPoolExecutor #357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c4c1e0c
Replace asyncio and httpx with urllib3 and ThreadPoolExecutor
hugovk Sep 15, 2026
6fd7acf
Update README tables with tox -e cog
hugovk Sep 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 25 additions & 23 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,32 +45,34 @@ Sphinx configuration in various branches:
| 3.10 | sphinx==3.4.3 | needs_sphinx='3.2' |
| 3.11 | sphinx~=7.2.0 | needs_sphinx='4.2' |
| 3.12 | sphinx~=8.2.0 | needs_sphinx='8.2.0' |
| 3.13 | sphinx~=8.2.0 | needs_sphinx='8.2.0' |
| 3.14 | sphinx~=8.2.0 | needs_sphinx='8.2.0' |
| 3.15 | sphinx~=8.2.0 | needs_sphinx='8.2.0' |
| 3.13 | sphinx<9.0.0 | needs_sphinx='8.2.0' |
| 3.14 | sphinx<9.0.0 | needs_sphinx='8.2.0' |
| 3.15 | sphinx<9.0.0 | needs_sphinx='8.2.0' |
| 3.16 | sphinx<9.0.0 | needs_sphinx='8.2.0' |

Sphinx build as seen on docs.python.org:

| version | el | en | es | fr | bn-in | id | it | ja | ko | pl | pt-br | ro | sv | tr | uk | zh-cn | zh-tw |
|-----------|-------|-------|-------|-------|---------|-------|-------|-------|-------|-------|---------|-------|-------|-------|-------|---------|---------|
| 2.6 | ø | 0.6.5 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 2.7 | ø | 2.3.1 | ø | 2.3.1 | ø | 2.3.1 | ø | 2.3.1 | 2.3.1 | ø | 2.3.1 | ø | ø | ø | ø | 2.3.1 | 2.3.1 |
| 3.0 | ø | 0.6 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.1 | ø | 0.6.5 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.2 | ø | 1.0.7 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.3 | ø | 1.2 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.4 | ø | 1.2.3 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.5 | ø | 1.8.4 | 1.8.4 | 1.8.4 | ø | 1.8.4 | ø | 1.8.4 | 1.8.4 | 1.8.4 | 1.8.4 | ø | ø | ø | ø | 1.8.4 | 1.8.4 |
| 3.6 | ø | 2.3.1 | 2.3.1 | 2.3.1 | ø | 2.3.1 | ø | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | ø | ø | ø | ø | 2.3.1 | 2.3.1 |
| 3.7 | ø | 2.3.1 | 2.3.1 | 2.3.1 | ø | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | ø | ø | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 |
| 3.8 | ø | 2.4.4 | 2.4.4 | 2.4.4 | ø | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | ø | ø | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 |
| 3.9 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | ø | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 |
| 3.10 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | ø | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 |
| 3.11 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | ø | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 |
| 3.12 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | ø | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.13 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.14 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.15 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| version | el | en | es | fr | bn-in | id | it | ja | ko | pl | pt-br | ro | ru | sv | tr | uk | zh-cn | zh-tw |
|-----------|-------|-------|-------|-------|---------|-------|-------|-------|-------|-------|---------|-------|-------|-------|-------|-------|---------|---------|
| 2.6 | ø | 0.6.5 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 2.7 | ø | 2.3.1 | ø | 2.3.1 | ø | 2.3.1 | ø | 2.3.1 | 2.3.1 | ø | 2.3.1 | ø | ø | ø | ø | ø | 2.3.1 | 2.3.1 |
| 3.0 | ø | 0.6 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.1 | ø | 0.6.5 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.2 | ø | 1.0.7 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.3 | ø | 1.2 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.4 | ø | 1.2.3 | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø | ø |
| 3.5 | ø | 1.8.4 | 1.8.4 | 1.8.4 | ø | 1.8.4 | ø | 1.8.4 | 1.8.4 | 1.8.4 | 1.8.4 | ø | ø | ø | ø | ø | 1.8.4 | 1.8.4 |
| 3.6 | ø | 2.3.1 | 2.3.1 | 2.3.1 | ø | 2.3.1 | ø | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | ø | ø | ø | ø | ø | 2.3.1 | 2.3.1 |
| 3.7 | ø | 2.3.1 | 2.3.1 | 2.3.1 | ø | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 | ø | ø | ø | 2.3.1 | 2.3.1 | 2.3.1 | 2.3.1 |
| 3.8 | ø | 2.4.4 | 2.4.4 | 2.4.4 | ø | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | ø | ø | ø | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 |
| 3.9 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | ø | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 | 2.4.4 |
| 3.10 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 | 3.4.3 |
| 3.11 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 | 7.2.6 |
| 3.12 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.13 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.14 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.15 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
| 3.16 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 | 8.2.3 |
<!-- [[[end]]] -->

Install `tools_requirements.txt` then run `python check_versions.py
Expand Down
40 changes: 21 additions & 19 deletions check_versions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
#!/usr/bin/env python

import argparse
import asyncio
import concurrent.futures
import logging
import re
from pathlib import Path

import git
import httpx
import urllib3
from tabulate import tabulate

import build_docs

logger = logging.getLogger(__name__)
http = urllib3.PoolManager()
MAX_WORKERS = 16
http = urllib3.PoolManager(maxsize=MAX_WORKERS)
VERSIONS = build_docs.parse_versions_from_peps_site(http)
LANGUAGES = build_docs.parse_languages_from_config()

Expand Down Expand Up @@ -87,33 +87,38 @@ def search_sphinx_versions_in_cpython(repo: git.Repo):
print(tabulate(table, headers=headers, tablefmt="github", disable_numparse=True))


async def get_version_in_prod(language: str, version: str) -> str:
def get_version_in_prod(language: str, version: str) -> str:
if language == "en":
url = f"https://docs.python.org/{version}/"
else:
url = f"https://docs.python.org/{language}/{version}/"
async with httpx.AsyncClient() as client:
try:
response = await client.get(url, timeout=5)
except httpx.ConnectTimeout:
return "(timeout)"
try:
response = http.request("GET", url, timeout=5)
except urllib3.exceptions.HTTPError:
return "(timeout)"
# Python 2.6--3.7: sphinx.pocoo.org
# from Python 3.8: www.sphinx-doc.org
if created_using := re.search(
r"(?:sphinx.pocoo.org|www.sphinx-doc.org).*?([0-9.]+[0-9])", response.text
r"(?:sphinx.pocoo.org|www.sphinx-doc.org).*?([0-9.]+[0-9])",
response.data.decode("utf-8", errors="replace"),
):
return created_using.group(1)
return "ø"


async def which_sphinx_is_used_in_production():
def which_sphinx_is_used_in_production():
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
futures = {
(version.name, language.tag): executor.submit(
get_version_in_prod, language.tag, version.name
)
for version in VERSIONS
for language in LANGUAGES
}
table = [
[
version.name,
*await asyncio.gather(*[
get_version_in_prod(language.tag, version.name)
for language in LANGUAGES
]),
*[futures[version.name, language.tag].result() for language in LANGUAGES],
]
for version in VERSIONS
]
Expand All @@ -123,15 +128,12 @@ async def which_sphinx_is_used_in_production():

def check_versions(cpython_clone: str) -> None:
logging.basicConfig(level=logging.INFO)
logging.getLogger("charset_normalizer").setLevel(logging.WARNING)
logging.getLogger("asyncio").setLevel(logging.WARNING)
logging.getLogger("httpx").setLevel(logging.WARNING)
repo = git.Repo(cpython_clone)
print("Sphinx configuration in various branches:", end="\n\n")
search_sphinx_versions_in_cpython(repo)
print()
print("Sphinx build as seen on docs.python.org:", end="\n\n")
asyncio.run(which_sphinx_is_used_in_production())
which_sphinx_is_used_in_production()


def main():
Expand Down
1 change: 0 additions & 1 deletion tools_requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
GitPython
httpx
tabulate
zc.lockfile