Skip to content

Use Python extension with libwcwidth - #248

Merged
jquast merged 20 commits into
masterfrom
jq/libwcwidth-shim-wheels
Sep 23, 2026
Merged

jquast merged 20 commits into
masterfrom
jq/libwcwidth-shim-wheels

Conversation

@jquast

@jquast jquast commented Sep 15, 2026 •

Copy link
Copy Markdown
Owner

This prepares a 0.9.0 release, which only enables use of C11 libwcwidth as optional Python extension. This PR makes no logical changes to the python library behavior except to improve performance ~10-58x by use of the optional build_ext.

This creates a new release process, documented in docs/developing.rst. tox -e check_release calls bin/check-release.py to help ensure all wheels are included.

A lot of care was done to make it safe to release to general public, that it should be possible to install from sdist even if compilation fails or is unavailable and gracefully fallback to pure python. cibuildwheel is used to build binaries, and, because we distribute a "python3-none-any" (any os, any architecture), that, if no build wheel exists, then package installs will use the python-only wheel without any error or warning.

@codecov

codecov Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (97b4df6) to head (77fda52).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #248   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           27        27           
  Lines         2008      2040   +32     
  Branches       468       473    +5     
=========================================
+ Hits          2008      2040   +32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed

codspeed Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×9.9

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 79 improved benchmarks
✅ 40 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_wcswidth_ri_flags_paired 342.7 ms 5.9 ms ×58
⚡ test_width_ri_flags_paired 342.9 ms 5.9 ms ×58
⚡ test_wcstwidth_ri_term_program[ghostty] 344 ms 6 ms ×58
⚡ test_width_ri_term_program[ghostty] 344.2 ms 6 ms ×57
⚡ test_wcswidth_ri_flags_unpaired 85.9 ms 1.5 ms ×57
⚡ test_wcstwidth_ri_term_program[xterm.js] 345.4 ms 6.1 ms ×57
⚡ test_width_ri_flags_unpaired 86 ms 1.5 ms ×57
⚡ test_width_ri_term_program[xterm.js] 345.6 ms 6.1 ms ×57
⚡ test_width_udhr 585.7 ms 13.7 ms ×43
⚡ test_clip_long_cjk_to_end 11,784.4 µs 278.9 µs ×42
⚡ test_clip_cjk_paragraph 11,709.6 µs 278.1 µs ×42
⚡ test_clip_dense_ansi_to_end 12,460 µs 303.1 µs ×41
⚡ test_clip_dense_ansi_to_end_no_propagate 10,114.5 µs 283.3 µs ×36
⚡ test_wcstwidth_emoji_term_program[ghostty] 1,373.6 µs 41.8 µs ×33
⚡ test_wcswidth_decomposed 2,600.1 µs 81.1 µs ×32
⚡ test_width_fastpath_integrity_udhr 739.7 ms 23.8 ms ×31
⚡ test_clip_dense_ansi_no_propagate 2,419.6 µs 86.1 µs ×28
⚡ test_width_udhr_lines 319.5 ms 12.1 ms ×26
⚡ test_wcswidth_composed 1,863.8 µs 70.7 µs ×26
⚡ test_clip_dense_ansi_no_propagate_no_overtype 2,191.7 µs 87 µs ×25
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jq/libwcwidth-shim-wheels (77fda52) with master (97b4df6)

Open in CodSpeed

@jquast
jquast changed the base branch from master to jq/unicode-18 September 15, 2026 13:19
@jquast
jquast added this pull request to stack #249 September 15, 2026 13:19
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch from fceb715 to 8056323 Compare September 15, 2026 19:12
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch from 8056323 to 5899da0 Compare September 15, 2026 21:26
jquast added a commit that referenced this pull request Sep 15, 2026
Adds sub-folder to this project, ``libwcwidth/``, a C11 implementation of the wcwidth API. This is a **release candidate**, relased by git tags on github along with matching version of python wcwidth, but **not yet included as a Python C extension**, this code is not distributed by pypi.

#248 creates the C Python "shims" to join the projects together.

Note that the C tables are generated from the **Unicode 18.0.0 (draft)**, 18.0 is expected to release later today, so we intend for these to match before any next python release or git version tag.
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch from 5899da0 to dd83e78 Compare September 15, 2026 21:54
Base automatically changed from jq/unicode-18 to master September 16, 2026 19:33
jquast added a commit that referenced this pull request Sep 16, 2026
We expect Unicode 18.0 to release today.

- Match new 18.0.0 ``UAX #20`` rules on grapheme clustering InCB/Indic_Conjunct_Break
  - Stop using python's ``iter_graphemes()``, reduces performance! fixed in next release #248 
- parse names from unicode.org data instead of current python
- `wcstwidth()` correction table updated with latest ucs-detect results (contour and rio are better-conforming)
- skip "bad results" from ucs-detect (delta_y movement in CPR, iTerm2, from correction tables)
- correction table file hashes are now deterministic (don't use repr() for hashing -- write our own)
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch from dd83e78 to 6bbbba7 Compare September 17, 2026 00:46
@jquast
jquast removed this pull request from stack #249 September 17, 2026 00:47
@jquast jquast changed the title (DRAFT) wcwidth 0.9.0rc1 uses C11 Python extension Use Python extension with libwcwidth Sep 17, 2026
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch 3 times, most recently from 225afb7 to 8bd4e2d Compare September 17, 2026 20:57
Comment thread tests/test_core.py Fixed
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch 3 times, most recently from d830610 to 84dbc43 Compare September 18, 2026 00:00
This will merge and release after release of wcwidth 0.8.4. The 0.9.0 release should
represent only the change of using C11 libwcwidth as optional Python extension. This
PR makes no logical change to wcwidth library, only that it drops Python 3.8 support
and improves performance by use of the optional build_ext.

This creates a new release process, documented in docs/developing.rst. tox -e
check_release calls bin/check-release.py to help ensure all wheels are included.

clip() keeps both Python implementations: _clip_simple() for text without cursor
movement, and _clip_painter() when movement must be resolved. Retiring
_clip_simple() in favor of a single painter cost 20-35% on seven clip()
benchmarks, and the extension does not recover it: libwcwidth declines OSC 8 and
OSC 66, and is never consulted for control_codes != 'parse', so those inputs are
answered by Python in either build. tests/test_clip.py compares the two paths
over the parity and fuzz corpora through the public overtyping= parameter.

A lot of care was done to make it safe to release to general public,
that it should be possible to install from sdist even if compilation
fails or is unavailable and gracefully fall back to pure python.
cibuildwheel is used to build binaries, and, because we distribute a
"py3-none-any" (any os, any architecture), that, if no build wheel
exists, then package installs will use the python-only wheel without any
error or warning.
@jquast
jquast force-pushed the jq/libwcwidth-shim-wheels branch from 84dbc43 to b36fc0c Compare September 18, 2026 00:09
@jquast
jquast marked this pull request as ready for review September 18, 2026 00:09
Comment thread wcwidth/_wcwidth_c.c
@jquast
jquast merged commit b7a3098 into master Sep 23, 2026
32 checks passed
@jquast
jquast deleted the jq/libwcwidth-shim-wheels branch September 23, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants