felis-terminal/felis

Watch

Join across a ZWJ only between pictographs; report ?2027 as permanently set #26

Merged

natsukium merged 2 commits from fix/width-cluster-rules into main

2026-10-04 20:57:36 +09:00

natsukium commented

2026-10-04 20:22:19 +09:00

Owner

Fixes the bug parts of #15.

GB11: a ZWJ now joins only a pictograph to a pictograph, with only Extend scalars between the pictograph and the joiner. 👩‍字, क‍ख‍ग and x‍👍 take 4, 3 and 3 cells (as wcwidth and kitty count) instead of collapsing into one two-cell glyph. A ZWSP or a bidi override that arrives after the base also breaks the join; an override held pending from before the base does not. Extended_Pictographic and Extend are Unicode 17 range tables, cross-checked over every scalar against unicode-segmentation (dev-dependency).
DECRQM ?2027 answers 3 (permanently set) instead of 1.

Not changed:

Emoji modifier after a non-base (a🏻b): already 3 cells on main, the same as wcwidth, and UAX#29 puts a🏻 in one cluster. Not a bug.
GB9c (Indic conjuncts) stays unfolded; REQ-602 states the exception and #25 tracks the decision.
U+00AD, U+3164/U+FFA0, Unicode version skew: policy items, left on #15.

Benches: ascii_throughput is 2–6% slower with byte-identical hot functions (print_ascii_bulk_gap, print_str). perf shows more branch misses at the same instruction mix, i.e. code placement. Putting the flag as a Grid field cost 14%, so it lives in PendingBidi's padding. unicode_throughput and scroll_region are within noise.

Fixes the bug parts of #15. - **GB11**: a ZWJ now joins only a pictograph to a pictograph, with only Extend scalars between the pictograph and the joiner. `👩‍字`, `क‍ख‍ग` and `x‍👍` take 4, 3 and 3 cells (as wcwidth and kitty count) instead of collapsing into one two-cell glyph. A ZWSP or a bidi override that arrives after the base also breaks the join; an override held pending from before the base does not. `Extended_Pictographic` and `Extend` are Unicode 17 range tables, cross-checked over every scalar against `unicode-segmentation` (dev-dependency). - **DECRQM `?2027`** answers `3` (permanently set) instead of `1`. Not changed: - Emoji modifier after a non-base (`a🏻b`): already 3 cells on main, the same as wcwidth, and UAX#29 puts `a🏻` in one cluster. Not a bug. - GB9c (Indic conjuncts) stays unfolded; REQ-602 states the exception and #25 tracks the decision. - U+00AD, U+3164/U+FFA0, Unicode version skew: policy items, left on #15. Benches: ascii_throughput is 2–6% slower with byte-identical hot functions (`print_ascii_bulk_gap`, `print_str`). perf shows more branch misses at the same instruction mix, i.e. code placement. Putting the flag as a `Grid` field cost 14%, so it lives in `PendingBidi`'s padding. unicode_throughput and scroll_region are within noise.

compare15.png

79 KiB

natsukium added 2 commits

2026-10-04 20:22:19 +09:00

grid: report grapheme cluster mode as permanently set 51ccdf6449

DECRST ?2027 has never taken effect: REQ-602 makes UAX#29 clustering
an invariant with no legacy path. Yet DECRQM answered `1` (set), which
tells an application the mode can be reset, so one that turns it off
for a region keeps believing the terminal now counts per scalar. `3`
(permanently set) is the answer the DECRQM Ps values reserve for that,
and the terminal-unicode-core spec asks for a reply that "indicates
each state accurately". Neovim's mode-report handler already treats 3
like 1, so detection keeps working.

The value lives in a PERMANENTLY_SET_DEC_MODES table beside the
permanently-reset one, checked first so a soft-tracked DECSET/DECRST
cannot shadow it, with a proptest mirroring the Ps=4 one.

Refs: #15
Assisted-by: Claude Code

grid: join across a ZWJ only between pictographs

bench / Criterion full-suite snapshot (pull_request) Has been skipped

Details

darwin / detect relevant changes (pull_request) Successful in 1m51s

Details

darwin / darwin docs-only no-op (pull_request) Has been skipped

Details

fuzz / cargo fuzz nightly long-run (pull_request) Has been skipped

Details

windows / detect relevant changes (pull_request) Successful in 1m55s

Details

windows / windows docs-only no-op (pull_request) Has been skipped

Details

fuzz / cargo fuzz smoke (per target) (pull_request) Successful in 1m46s

Details

windows / cargo nextest (Windows) (pull_request) Successful in 4m34s

Details

pr / nix flake check (pull_request) Successful in 1m54s

Details

windows / frontend smoke (Windows) (pull_request) Successful in 1m8s

Details

windows / package felis (x86_64-pc-windows-msvc) (pull_request) Has been skipped

Details

pr / frontend smoke (x86_64-linux) (pull_request) Successful in 1m19s

Details

bench / Criterion regression gate (pull_request) Successful in 2m29s

Details

darwin / build felis (aarch64-darwin) (pull_request) Successful in 6m16s

Details

pr / cargo build / clippy / test / deny (pull_request) Successful in 3m7s

Details

pr / wire schema is compatible with the base (pull_request) Successful in 34s

Details

windows / cargo clippy (Windows cross) (pull_request) Successful in 44s

Details

e2f0c08e2b

The GB11 continuation folded any non-ASCII scalar printed after a
cluster ending in ZWJ, so `👩‍字`, `क‍ख‍ग` and `x‍👍` collapsed into
one cluster that grapheme_width's two-cell cap squeezed to 2 cells.
UAX#29 breaks all three, and applications (wcwidth sums and kitty
alike) count 4, 3 and 3, so the cursor desynced and the rest of the
line was drawn left of where the application put it.

The join now follows GB11 itself: an Extended_Pictographic follower,
after a joiner that only Extend scalars separate from an
Extended_Pictographic. The width table cannot stand in for either
side: the fold admits zero-width scalars that UAX#29 breaks on (ZWSP,
the bidi overrides), so felis carries both properties as Unicode 17
range tables (the version unicode-width encodes), cross-checked over
every scalar against unicode-segmentation as a dev-dependency.

A bidi override folded after the base is a break, while one held
pending from before the base is not, and the two look the same in the
stored cluster; `PendingBidi::after_base` records which one arrived,
even when a full cluster table refuses the fold. It lives on
PendingBidi because one more Grid field measured 14% slower on the
ASCII print benches (the hot loop is unchanged; code placement moved).

GB9c (Indic conjuncts) stays unfolded: a conjunct chain outgrows the
two-cell cap and the terminals that fold it disagree on its width.
REQ-602 now states the exception, and #25 tracks the decision. The
ZWJ half-form `क्‍ख` therefore lands as two cells like `क्ख` already
did, at the same width as before.

Closes the GB11 and DECRQM items of #15; the emoji-modifier item is not
a bug (`a🏻b` is already 3 cells, as wcwidth counts).

Refs: #15
Assisted-by: Claude Code