natsukium/felis
Archived
Watch
1
Code Issues 23 Pull requests Projects Releases 6 Packages Wiki Activity Actions

daemon: Linux vtebench scrolling rows trail the peers and vary 25-50% between rounds #400

Open
opened 2026-09-25 01:34:55 +09:00 by natsukium · 6 comments
natsukium commented 2026-09-25 01:34:55 +09:00
Owner
Copy link

On Linux felis trails on every vtebench scrolling row except scrolling_fullscreen and varies 25-50% between rounds (docs/reference/benchmarks.md, felis 9d1d842f):

Row felis foot alacritty ghostty-tip
scrolling 145 118 128 288
scrolling_bottom_region 150 122 121 111
scrolling_top_region 144 125 127 112

On macOS the same rows take felis 9.5-10.1 ms and it leads them. Every terminal on Linux sits above 100 ms, so the gap is likely not the parser: something between the PTY and the daemon's read loop sets the pace on Linux.

Plan:

  1. Count the daemon's PTY reads and bytes per read during these payloads on Linux and on macOS.
  2. Check whether the read loop waits on a paint tick, a lock hand-off (#380), or a channel between the reader thread and the parser.
  3. Measure with the powersave governor noted in the run conditions, and with the load average recorded.

Done when felis's median on these rows is at or below the Linux peers', or the cause is shown to be outside felis and recorded.

On Linux felis trails on every vtebench scrolling row except `scrolling_fullscreen` and varies 25-50% between rounds (`docs/reference/benchmarks.md`, felis `9d1d842f`): | Row | felis | foot | alacritty | ghostty-tip | |---|---|---|---|---| | scrolling | 145 | 118 | 128 | 288 | | scrolling_bottom_region | 150 | 122 | 121 | 111 | | scrolling_top_region | 144 | 125 | 127 | 112 | On macOS the same rows take felis 9.5-10.1 ms and it leads them. Every terminal on Linux sits above 100 ms, so the gap is likely not the parser: something between the PTY and the daemon's read loop sets the pace on Linux. Plan: 1. Count the daemon's PTY reads and bytes per read during these payloads on Linux and on macOS. 2. Check whether the read loop waits on a paint tick, a lock hand-off (#380), or a channel between the reader thread and the parser. 3. Measure with the `powersave` governor noted in the run conditions, and with the load average recorded. Done when felis's median on these rows is at or below the Linux peers', or the cause is shown to be outside felis and recorded.
natsukium commented 2026-09-25 02:47:23 +09:00
Author
Owner
Copy link

Linux measurement (kilimanjaro, i5-12400F, niri 60 Hz, NVIDIA via a pinned ICD, see #398). felis main ef96bb46 release. Load average (1 min) is given with each figure; CI ran on the host throughout, so wall-clock medians swing by tens of percent between runs, while read counts and CPU splits are stable.

What paces the scrolling rows on Linux: the producer's own write path

  • vtebench itself spends 100% of the pass in the kernel. Wrapped in GNU time inside felis, it shows sys 10.07 s of 10.12 s wall, 5-78 voluntary context switches, user 0.00 s. It never blocks on a full PTY. It is CPU-bound in write(2), where the slave's line discipline expands every \n (ONLCR) and pushes each piece through pty_write → flip buffer → flush_to_ldisc.
  • No terminal involved: a Python consumer that only drains the master with blocking 64 KiB reads (ptyfloor.py, 35x106 winsize) gets 120-130 ms median on scrolling for the same 1.6 MB sample (load 3.8-4.5), with the writer's sys again equal to wall. That is the Linux floor for this payload, and it explains why every Linux terminal sits above 100 ms while macOS runs the same row in 10 ms.
  • felis is at or below that floor in manual runs: 81-117 ms median (188x93 and 141x53 windows, loads 3.7-11). The published 144-150 ms (and 142-145 ms in my 106x35 harness rerun at load 2.6-4.9) sit above it, but the foot figures from the same harness swing from 80 to 163 ms between two runs at loads 8-17. I could not get a quiet enough host to size felis-vs-foot at the harness grid more precisely.

How the daemon reads (the question in the plan)

run load reads bytes bytes/read reader user+sys reader vol. ctx sw
felis scrolling 12 s, 188x93 8.8 14.0 M 178 MB 13 0.75 + 9.01 s 184 k
felis scrolling 10 s, 141x53 3.7 12.0 M 159 MB 13 0.87 + 7.61 s 189 k
foot scrolling 10 s, 282x108 7.9 649 k 88 MB 136 8.63 + 0.57 s (whole foot main thread) 15 k
no-op Python drain, 35x106 3.8 6.9 M 98 MB 14 3.76 + 3.90 s -

(/proc/<pid>/task/<tid>/io syscr/rchar, read at the pass's start and end markers.)

  • The read loop does not wait on a paint tick, the parse lock, or the channel. The parser thread parks about 8,000 times a second and is woken promptly. The reader sees 13 bytes each time because it is always caught up with the writer. Every read(2) returns the few lines the writer has pushed since the previous one, about 1.2 M syscalls/s.
  • The 50 µs EAGAIN spin in MasterReader::read (crates/felis-pty/src/unix.rs:67-104) is not what drives the count here. With the spin set to 0 the pass still makes 13-15 M reads of 12-16 bytes. On DOOM-fire the spin does cost reader user time (#399).
  • foot reads 10x more per call only because its single thread parses and renders between reads.

Experiment: coalesce small reads (local, env-gated patch, not committed)

After a read shorter than 2048 bytes, and only when the parser was not parked, pump_reads (crates/felis-pty/src/lib.rs:634) sleeps 20 µs before the next read:

run load median bytes/read reader user+sys
baseline, 141x53 3.7 / 6.1 90.5 / 95.0 ms 13 / 16 8.48 / 7.50 s
back-off, 141x53 4.1 / 6.8 80.5 / 81.0 ms 566 / 787 0.67 / 0.64 s
baseline, 188x93 8.8 / 8.4 101.5 / 89.0 ms 13 / 14 9.76 / 9.91 s
back-off, 188x93 8.0 / 7.8 78.0 / 81.0 ms 869 / 672 0.76 / 0.76 s
harness vtebench (all rows, 106x35): baseline 8.6 → 17.2 scrolling 100.5, top 92.0, bottom 88.5 ms - daemon 83.1 s
harness vtebench: back-off on by default 17.2 → 8.4 scrolling 103.0, top 105.0, bottom 92.0 ms - daemon 55.9 s

On scrolling, reader CPU drops by 92%, and the median drops by 10-20% in the manual pairs. In the harness pair the rows are flat within noise, and daemon CPU for the whole vtebench pass drops by 33%.

The back-off as written is not shippable. Under DOOM-fire it starves the parser: the frame rate fell from 933 / 795 fps to 400 / 666 fps with a 2048-byte gate, and 929 → 601 fps with a 256-byte gate in one of two pairs. One PTY read on Linux never returns more than about 4 KiB (the N_TTY_BUF_SIZE read buffer), so every sleep caps ingest.

Proposed fix

Coalesce reads in pump_reads only while the parser still has a backlog. The reader backs off after a small read only when filling already holds at least MIN_SWAP (8 KiB, crates/felis-pty/src/lib.rs:726) that the parser has not taken yet, or it waits on poll(2) with a short timeout instead of sleep. Keystroke echo is unaffected, because an idle parser is parked and the back-off does not apply. Expected gain on Linux: reader CPU down about 90% on line-heavy output (7-10 s per 10-12 s pass), daemon CPU on vtebench down about 30%, scrolling rows down 0-20%. DOOM-fire frame rate must stay unchanged; any patch needs a DOOM-fire A/B on a quiet host. Separately, dropping or shortening the 50 µs spin on Linux saves 1.4-1.7 s of reader user time per 12 s of DOOM-fire (#399).

On the done condition: the >100 ms level on Linux is set outside felis, by the kernel's PTY output processing in the producer's context (the no-op drain floor is 120-130 ms). What felis controls is the syscall rate of its reader thread; the fix above cuts that rate by more than 90%.

Linux measurement (kilimanjaro, i5-12400F, niri 60 Hz, NVIDIA via a pinned ICD, see #398). felis `main` ef96bb46 release. Load average (1 min) is given with each figure; CI ran on the host throughout, so wall-clock medians swing by tens of percent between runs, while read counts and CPU splits are stable. ## What paces the scrolling rows on Linux: the producer's own write path - `vtebench` itself spends 100% of the pass in the kernel. Wrapped in GNU `time` inside felis, it shows `sys` 10.07 s of 10.12 s wall, 5-78 voluntary context switches, `user` 0.00 s. It never blocks on a full PTY. It is CPU-bound in `write(2)`, where the slave's line discipline expands every `\n` (ONLCR) and pushes each piece through `pty_write` → flip buffer → `flush_to_ldisc`. - No terminal involved: a Python consumer that only drains the master with blocking 64 KiB reads (`ptyfloor.py`, 35x106 winsize) gets **120-130 ms** median on `scrolling` for the same 1.6 MB sample (load 3.8-4.5), with the writer's `sys` again equal to wall. That is the Linux floor for this payload, and it explains why every Linux terminal sits above 100 ms while macOS runs the same row in 10 ms. - felis is at or below that floor in manual runs: 81-117 ms median (188x93 and 141x53 windows, loads 3.7-11). The published 144-150 ms (and 142-145 ms in my 106x35 harness rerun at load 2.6-4.9) sit above it, but the foot figures from the same harness swing from 80 to 163 ms between two runs at loads 8-17. I could not get a quiet enough host to size felis-vs-foot at the harness grid more precisely. ## How the daemon reads (the question in the plan) | run | load | reads | bytes | bytes/read | reader user+sys | reader vol. ctx sw | |---|---|---|---|---|---|---| | felis `scrolling` 12 s, 188x93 | 8.8 | 14.0 M | 178 MB | 13 | 0.75 + 9.01 s | 184 k | | felis `scrolling` 10 s, 141x53 | 3.7 | 12.0 M | 159 MB | 13 | 0.87 + 7.61 s | 189 k | | foot `scrolling` 10 s, 282x108 | 7.9 | 649 k | 88 MB | 136 | 8.63 + 0.57 s (whole foot main thread) | 15 k | | no-op Python drain, 35x106 | 3.8 | 6.9 M | 98 MB | 14 | 3.76 + 3.90 s | - | (`/proc/<pid>/task/<tid>/io` `syscr`/`rchar`, read at the pass's start and end markers.) - The read loop does not wait on a paint tick, the parse lock, or the channel. The parser thread parks about 8,000 times a second and is woken promptly. The reader sees 13 bytes each time because it is always caught up with the writer. Every `read(2)` returns the few lines the writer has pushed since the previous one, about 1.2 M syscalls/s. - The 50 µs `EAGAIN` spin in `MasterReader::read` (`crates/felis-pty/src/unix.rs:67-104`) is not what drives the count here. With the spin set to 0 the pass still makes 13-15 M reads of 12-16 bytes. On DOOM-fire the spin does cost reader user time (#399). - foot reads 10x more per call only because its single thread parses and renders between reads. ## Experiment: coalesce small reads (local, env-gated patch, not committed) After a read shorter than 2048 bytes, and only when the parser was not parked, `pump_reads` (`crates/felis-pty/src/lib.rs:634`) sleeps 20 µs before the next read: | run | load | median | bytes/read | reader user+sys | |---|---|---|---|---| | baseline, 141x53 | 3.7 / 6.1 | 90.5 / 95.0 ms | 13 / 16 | 8.48 / 7.50 s | | back-off, 141x53 | 4.1 / 6.8 | 80.5 / 81.0 ms | 566 / 787 | 0.67 / 0.64 s | | baseline, 188x93 | 8.8 / 8.4 | 101.5 / 89.0 ms | 13 / 14 | 9.76 / 9.91 s | | back-off, 188x93 | 8.0 / 7.8 | 78.0 / 81.0 ms | 869 / 672 | 0.76 / 0.76 s | | harness vtebench (all rows, 106x35): baseline | 8.6 → 17.2 | scrolling 100.5, top 92.0, bottom 88.5 ms | - | daemon 83.1 s | | harness vtebench: back-off on by default | 17.2 → 8.4 | scrolling 103.0, top 105.0, bottom 92.0 ms | - | daemon **55.9 s** | On `scrolling`, reader CPU drops by 92%, and the median drops by 10-20% in the manual pairs. In the harness pair the rows are flat within noise, and daemon CPU for the whole vtebench pass drops by 33%. The back-off as written is not shippable. Under DOOM-fire it starves the parser: the frame rate fell from 933 / 795 fps to 400 / 666 fps with a 2048-byte gate, and 929 → 601 fps with a 256-byte gate in one of two pairs. One PTY read on Linux never returns more than about 4 KiB (the `N_TTY_BUF_SIZE` read buffer), so every sleep caps ingest. ## Proposed fix Coalesce reads in `pump_reads` only while the parser still has a backlog. The reader backs off after a small read only when `filling` already holds at least `MIN_SWAP` (8 KiB, `crates/felis-pty/src/lib.rs:726`) that the parser has not taken yet, or it waits on `poll(2)` with a short timeout instead of `sleep`. Keystroke echo is unaffected, because an idle parser is parked and the back-off does not apply. Expected gain on Linux: reader CPU down about 90% on line-heavy output (7-10 s per 10-12 s pass), daemon CPU on vtebench down about 30%, scrolling rows down 0-20%. DOOM-fire frame rate must stay unchanged; any patch needs a DOOM-fire A/B on a quiet host. Separately, dropping or shortening the 50 µs spin on Linux saves 1.4-1.7 s of reader user time per 12 s of DOOM-fire (#399). On the done condition: the >100 ms level on Linux is set outside felis, by the kernel's PTY output processing in the producer's context (the no-op drain floor is 120-130 ms). What felis controls is the syscall rate of its reader thread; the fix above cuts that rate by more than 90%.
natsukium commented 2026-09-25 07:07:34 +09:00
Author
Owner
Copy link

I tried the proposed fix, coalescing reads while the parser has a backlog. It does not hold up: it trades the reader's CPU for the producer's throughput. #421 lands only the spin removal (#399) and records the rejection in docs/explanation/implementation.md.

Linux, i5-12400F, release builds, interleaved runs, 1-min load 2-6 unless noted:

  • Unread output slows the producer's write path. I used ptyfloor.py, a bare Python drain of the PTY master, on vtebench scrolling at 82x212:
    • Five interleaved pairs, reading back-to-back vs sleeping 50 µs between reads: 173 / 183.5 / 160 / 159 / 158.5 ms per sample vs 263 / 249.5 / 268 / 254 / 260 ms.
    • A busy-wait pause instead of a sleep, four rounds: 0 µs gave 129-160 ms, 20 µs 163-256 ms, 50 µs 214-256 ms.
    • The writer is at 100% sys in every case. It gets through fewer bytes when output sits unread, not because it blocks, and the consumer's sleep state is not the cause.
    • The same held at 93x212, 53x141 and 35x106 (one run each: 150-180 ms read back-to-back, 223-253 ms with a 50 µs pause).
  • felis with a coalescing reader. Only on Linux: after a read, wait up to 50 µs on a condvar while the parser is not parked. The wait is cut short when the parser parks, and it applies only while fewer than 2 KiB arrive per 50 µs and no read hit the 4 KiB line-discipline cap.
    • Reader on scrolling at 93x212: 13 → 650-920 bytes per read, 7.8 s → 0.55 s of reader CPU per 10 s.
    • DOOM-fire fps unchanged (979 vs 970).
    • Harness vtebench at 212x82 went bimodal: scrolling_top_region and scrolling_top_small_region medians 212 / 231 ms vs 105 / 107 ms on main, with samples split between ~80-100 ms and ~220-245 ms. A second run put every scrolling row at 215-235 ms.
    • Instrumented reads in the slow mode showed the producer delivering ~7 MB/s while the reader waited 96% of the time. That matches the floor above.
  • Harness daemon CPU on vtebench was 128 s (main) vs 84-90 s (coalescing), load 4.6-6.5.

So what felis controls on these rows is how promptly the reader drains, and draining promptly is what keeps the producer fast. The scrolling rows' CPU is the price of their throughput.

I have not sized the harness gap to foot (the published 144-150 vs 118-125 ms). In #421's interleaved manual pairs, the spin removal is neutral or better on these rows: scrolling 130.5 / 123 / 133 → 126 / 113 / 108.5 ms. Leaving this open for that call.

I tried the proposed fix, coalescing reads while the parser has a backlog. It does not hold up: it trades the reader's CPU for the producer's throughput. #421 lands only the spin removal (#399) and records the rejection in `docs/explanation/implementation.md`. Linux, i5-12400F, release builds, interleaved runs, 1-min load 2-6 unless noted: - **Unread output slows the producer's write path.** I used `ptyfloor.py`, a bare Python drain of the PTY master, on vtebench `scrolling` at 82x212: - Five interleaved pairs, reading back-to-back vs sleeping 50 µs between reads: 173 / 183.5 / 160 / 159 / 158.5 ms per sample vs 263 / 249.5 / 268 / 254 / 260 ms. - A busy-wait pause instead of a sleep, four rounds: 0 µs gave 129-160 ms, 20 µs 163-256 ms, 50 µs 214-256 ms. - The writer is at 100% sys in every case. It gets through fewer bytes when output sits unread, not because it blocks, and the consumer's sleep state is not the cause. - The same held at 93x212, 53x141 and 35x106 (one run each: 150-180 ms read back-to-back, 223-253 ms with a 50 µs pause). - **felis with a coalescing reader.** Only on Linux: after a read, wait up to 50 µs on a condvar while the parser is not parked. The wait is cut short when the parser parks, and it applies only while fewer than 2 KiB arrive per 50 µs and no read hit the 4 KiB line-discipline cap. - Reader on `scrolling` at 93x212: 13 → 650-920 bytes per read, 7.8 s → 0.55 s of reader CPU per 10 s. - DOOM-fire fps unchanged (979 vs 970). - Harness vtebench at 212x82 went bimodal: `scrolling_top_region` and `scrolling_top_small_region` medians 212 / 231 ms vs 105 / 107 ms on main, with samples split between ~80-100 ms and ~220-245 ms. A second run put every scrolling row at 215-235 ms. - Instrumented reads in the slow mode showed the producer delivering ~7 MB/s while the reader waited 96% of the time. That matches the floor above. - **Harness daemon CPU on vtebench** was 128 s (main) vs 84-90 s (coalescing), load 4.6-6.5. So what felis controls on these rows is how promptly the reader drains, and draining promptly is what keeps the producer fast. The scrolling rows' CPU is the price of their throughput. I have not sized the harness gap to foot (the published 144-150 vs 118-125 ms). In #421's interleaved manual pairs, the spin removal is neutral or better on these rows: `scrolling` 130.5 / 123 / 133 → 126 / 113 / 108.5 ms. Leaving this open for that call.
natsukium commented 2026-09-25 16:07:03 +09:00
Author
Owner
Copy link

Measured at 64a5eb24 (nix build .#felis) against foot 1.28.0 (nix develop .#bench). I ran the harness vtebench suite (crossterm.py run --terminals foot vtebench, ROUNDS=3, leg order rotated) on kilimanjaro: i5-12400F, powersave governor, niri. The grid was 212x82 (DejaVu Sans Mono 10 pt) for every leg. Each value is the harness's per-leg mean in ms; lower is better.

Before each round the harness was held until the 1-minute load was below 2.5 and no Forgejo Actions task was running. The load at each round's start was 0.68, 2.39 and 2.29. With nothing else running, a vtebench leg by itself lifts the 1-minute load to about 3.5; the peak inside any leg was 3.6-4.2.

Row felis r1 / r2 / r3 felis median foot r1 / r2 / r3 foot median felis - foot
scrolling 118.1 / 144.7 / 131.5 131.5 134.4 / 132.6 / 137.6 134.4 -2.9 (ranges overlap)
scrolling_bottom_region 132.2 / 127.7 / 133.4 132.2 99.8 / 119.3 / 118.2 118.2 +14.0 (+12%), no overlap
scrolling_bottom_small_region 139.0 / 139.7 / 134.7 139.0 125.3 / 121.5 / 116.0 121.5 +17.5 (+14%), no overlap
scrolling_fullscreen 12.7 / 12.7 / 12.8 12.7 7.9 / 7.8 / 7.9 7.9 +4.8 (+61%), no overlap
scrolling_top_region 123.8 / 144.9 / 133.1 133.1 108.4 / 114.1 / 111.8 111.8 +21.3 (+19%), no overlap
scrolling_top_small_region 125.2 / 143.9 / 135.2 135.2 121.7 / 127.1 / 118.9 121.7 +13.5 (+11%), ranges overlap

Round orders: felis→foot, foot→felis, felis→foot.

  • felis's median is at or below foot's only on scrolling. scrolling_top_small_region overlaps foot's range, by a single round (felis 125.2 against foot 127.1).
  • felis still trails on four rows, with no overlap between the rounds: scrolling_bottom_region by 14 ms, scrolling_bottom_small_region by 17.5 ms, scrolling_top_region by 21 ms and scrolling_fullscreen by 4.8 ms.
  • At 212x82, scrolling_fullscreen takes felis 12.7 ms in every round, against foot's 7.9 ms.
  • Across rounds, felis's region rows vary by 5-21 ms and foot's by 6-20 ms.

The harness flagged one leg in each of rounds 2 and 3 as "unpinned (no window of this launch is mapped)": foot in round 2, felis in round 3. Both still ran at 212x82 cells and 1696 px wide, and their figures are in line with the pinned legs.

A second three-round run of the same build, started earlier the same afternoon, is discarded under the load rule. Rounds 2 and 3 started at load 3.8 and 6.3, and a CI cargo build lifted the load to 23 during foot's round-3 leg. Its medians had felis at or below foot on every scrolling row except scrolling_fullscreen (13.4 against 8.0 ms). On the region rows the two runs differ by about as much as the felis-foot gap in the run above.

Measured at `64a5eb24` (`nix build .#felis`) against foot 1.28.0 (`nix develop .#bench`). I ran the harness vtebench suite (`crossterm.py run --terminals foot vtebench`, `ROUNDS=3`, leg order rotated) on kilimanjaro: i5-12400F, `powersave` governor, niri. The grid was 212x82 (DejaVu Sans Mono 10 pt) for every leg. Each value is the harness's per-leg mean in ms; lower is better. Before each round the harness was held until the 1-minute load was below 2.5 and no Forgejo Actions task was running. The load at each round's start was 0.68, 2.39 and 2.29. With nothing else running, a vtebench leg by itself lifts the 1-minute load to about 3.5; the peak inside any leg was 3.6-4.2. | Row | felis r1 / r2 / r3 | felis median | foot r1 / r2 / r3 | foot median | felis - foot | |---|---|---|---|---|---| | scrolling | 118.1 / 144.7 / 131.5 | 131.5 | 134.4 / 132.6 / 137.6 | 134.4 | -2.9 (ranges overlap) | | scrolling_bottom_region | 132.2 / 127.7 / 133.4 | 132.2 | 99.8 / 119.3 / 118.2 | 118.2 | +14.0 (+12%), no overlap | | scrolling_bottom_small_region | 139.0 / 139.7 / 134.7 | 139.0 | 125.3 / 121.5 / 116.0 | 121.5 | +17.5 (+14%), no overlap | | scrolling_fullscreen | 12.7 / 12.7 / 12.8 | 12.7 | 7.9 / 7.8 / 7.9 | 7.9 | +4.8 (+61%), no overlap | | scrolling_top_region | 123.8 / 144.9 / 133.1 | 133.1 | 108.4 / 114.1 / 111.8 | 111.8 | +21.3 (+19%), no overlap | | scrolling_top_small_region | 125.2 / 143.9 / 135.2 | 135.2 | 121.7 / 127.1 / 118.9 | 121.7 | +13.5 (+11%), ranges overlap | Round orders: felis→foot, foot→felis, felis→foot. - felis's median is at or below foot's only on `scrolling`. `scrolling_top_small_region` overlaps foot's range, by a single round (felis 125.2 against foot 127.1). - felis still trails on four rows, with no overlap between the rounds: `scrolling_bottom_region` by 14 ms, `scrolling_bottom_small_region` by 17.5 ms, `scrolling_top_region` by 21 ms and `scrolling_fullscreen` by 4.8 ms. - At 212x82, `scrolling_fullscreen` takes felis 12.7 ms in every round, against foot's 7.9 ms. - Across rounds, felis's region rows vary by 5-21 ms and foot's by 6-20 ms. The harness flagged one leg in each of rounds 2 and 3 as "unpinned (no window of this launch is mapped)": foot in round 2, felis in round 3. Both still ran at 212x82 cells and 1696 px wide, and their figures are in line with the pinned legs. A second three-round run of the same build, started earlier the same afternoon, is discarded under the load rule. Rounds 2 and 3 started at load 3.8 and 6.3, and a CI cargo build lifted the load to 23 during foot's round-3 leg. Its medians had felis at or below foot on every scrolling row except `scrolling_fullscreen` (13.4 against 8.0 ms). On the region rows the two runs differ by about as much as the felis-foot gap in the run above.
natsukium commented 2026-09-25 18:17:50 +09:00
Author
Owner
Copy link

Full cross-terminal run at 5f5e4957 (nix build .#felis, which includes #447's scroll directives), on kilimanjaro: i5-12400F, powersave governor, niri, 106x35 cells, DejaVu Sans Mono 14 pt, the published page's workloads, ROUNDS=3 with the leg order rotated. The CI runner was stopped for the whole run; the Windows CI guest stayed up. Every leg completed in every round. Each value is the harness's per-leg mean in ms; lower is better.

Row felis r1 / r2 / r3 felis median foot r1 / r2 / r3 foot median felis - foot
scrolling 144.6 / 135.5 / 137.5 137.5 111.0 / 116.4 / 112.7 112.7 +24.8 (+22%), no overlap
scrolling_bottom_region 142.2 / 150.6 / 149.0 149.0 113.9 / 119.2 / 115.9 115.9 +33.1 (+29%), no overlap
scrolling_bottom_small_region 143.0 / 139.8 / 138.5 139.8 109.8 / 113.6 / 113.5 113.5 +26.3 (+23%), no overlap
scrolling_fullscreen 6.2 / 6.1 / 6.2 6.2 7.0 / 6.8 / 6.8 6.8 -0.6 (-9%), no overlap
scrolling_top_region 152.8 / 142.3 / 131.1 142.3 114.6 / 111.2 / 109.6 111.2 +31.1 (+28%), no overlap
scrolling_top_small_region 144.6 / 140.8 / 154.9 144.6 117.6 / 110.8 / 111.6 111.6 +33.0 (+30%), no overlap

Round orders: felis first in round 1, felis after foot in rounds 2 and 3.

  • scrolling_fullscreen is fixed at this grid: felis 6.2 ms against foot's 6.8 ms, ranges apart (at 9d1d842f it was 6.1 against 7.0; the 212x82 run above had felis 12.7 against 7.9, not re-measured here).
  • On the other five scrolling rows felis trails foot by 25-33 ms with no overlap in any row. alacritty (123-135 ms) is also below felis on four of the five, and ghostty-tip (109-119 ms) on the four region rows.
  • felis's round-to-round spread on these rows is 3-15% now, against 25-50% at 9d1d842f.
  • Load: 1.61 at the start. foot's legs followed ghostty-tip's in every round, whose vtebench legs lift the one-minute average to 10-14, so foot started each leg at 10.6 / 13.9 / 11.4 against felis's 1.6 / 3.6 / 3.5. The gap is against foot running under the higher average.

The done condition (median at or below the Linux peers') is not met on this quiet run, so this stays open. The earlier finding still bounds what felis controls: the >100 ms level is the producer's PTY write path, and the reader already drains promptly. The remaining 25-33 ms gap to foot is not explained by that floor.

The republished page is in PR #456.

Full cross-terminal run at `5f5e4957` (`nix build .#felis`, which includes #447's scroll directives), on kilimanjaro: i5-12400F, `powersave` governor, niri, 106x35 cells, DejaVu Sans Mono 14 pt, the published page's workloads, `ROUNDS=3` with the leg order rotated. The CI runner was stopped for the whole run; the Windows CI guest stayed up. Every leg completed in every round. Each value is the harness's per-leg mean in ms; lower is better. | Row | felis r1 / r2 / r3 | felis median | foot r1 / r2 / r3 | foot median | felis - foot | |---|---|---|---|---|---| | scrolling | 144.6 / 135.5 / 137.5 | 137.5 | 111.0 / 116.4 / 112.7 | 112.7 | +24.8 (+22%), no overlap | | scrolling_bottom_region | 142.2 / 150.6 / 149.0 | 149.0 | 113.9 / 119.2 / 115.9 | 115.9 | +33.1 (+29%), no overlap | | scrolling_bottom_small_region | 143.0 / 139.8 / 138.5 | 139.8 | 109.8 / 113.6 / 113.5 | 113.5 | +26.3 (+23%), no overlap | | scrolling_fullscreen | 6.2 / 6.1 / 6.2 | 6.2 | 7.0 / 6.8 / 6.8 | 6.8 | -0.6 (-9%), no overlap | | scrolling_top_region | 152.8 / 142.3 / 131.1 | 142.3 | 114.6 / 111.2 / 109.6 | 111.2 | +31.1 (+28%), no overlap | | scrolling_top_small_region | 144.6 / 140.8 / 154.9 | 144.6 | 117.6 / 110.8 / 111.6 | 111.6 | +33.0 (+30%), no overlap | Round orders: felis first in round 1, felis after foot in rounds 2 and 3. - `scrolling_fullscreen` is fixed at this grid: felis 6.2 ms against foot's 6.8 ms, ranges apart (at 9d1d842f it was 6.1 against 7.0; the 212x82 run above had felis 12.7 against 7.9, not re-measured here). - On the other five scrolling rows felis trails foot by 25-33 ms with no overlap in any row. alacritty (123-135 ms) is also below felis on four of the five, and ghostty-tip (109-119 ms) on the four region rows. - felis's round-to-round spread on these rows is 3-15% now, against 25-50% at 9d1d842f. - Load: 1.61 at the start. foot's legs followed ghostty-tip's in every round, whose vtebench legs lift the one-minute average to 10-14, so foot started each leg at 10.6 / 13.9 / 11.4 against felis's 1.6 / 3.6 / 3.5. The gap is against foot running under the higher average. The done condition (median at or below the Linux peers') is not met on this quiet run, so this stays open. The earlier finding still bounds what felis controls: the >100 ms level is the producer's PTY write path, and the reader already drains promptly. The remaining 25-33 ms gap to foot is not explained by that floor. The republished page is in PR #456.
natsukium commented 2026-09-25 20:03:14 +09:00
Author
Owner
Copy link

Profiling at origin/main 5f5e4957, headless sway at 106x35 and 14 pt, load 2-14 with the CI runner active. The load for every figure is in the scratch logs.

The Linux vtebench scrolling rows are bound by where the scheduler places threads, not by felis's pipeline.

  • A bare PTY drain spans 77-260 ms. The drain is a forkpty that reads back to back with no parsing, and its time depends on where the writer, the reader and the unbound kworker running flush_to_ldisc land:

    • flush worker on the writer's SMT sibling: 77-89 ms
    • all three on separate cores: about 130-150 ms
    • reader on the writer's sibling: 178-260 ms

  • The placement is chance. The worker stays on its previous CPU while that CPU is idle, and otherwise prefers a fully idle core.

  • felis is fastest when the placement is favorable. With the terminal pinned to CPUs 0-7 and vtebench to CPU 10, felis scored 82-103 ms, with the worker next to the writer 91-94% of the time. foot scored 93-206 ms in the same setup, with the worker next to the writer 51% of the time. Unpinned, felis scored 111-136 ms and foot 94-111 ms.

  • Some gap remains at equal placement. In two rounds where the placement was about the same, foot scored 104-107 ms and felis 120-128 ms. felis's reader makes about 1.2 M reads/s at about 10 B each, while foot's main thread makes about 0.4 M reads/s at about 38 B each.

  • Levers tried without a clear gain:

    • a 50 µs read spin
    • backing the parser off 1-4 ms while output streams
    • a 3 µs busy pause after short reads: about 8% better, within noise, and it moves kernel time into user-space spinning

    A reader that sleeps between reads still slows the producer.

Next steps:

  • Confirm the mechanism with a kernel profile (perf c2c, or perf record -a -g on the writer and the flush worker in the three layouts). This needs perf_event_paranoid ≤ 1.
  • In the benchmark methodology, treat these rows as placement-bound: publish from many interleaved rounds, or add a pinned-layout variant.
Profiling at origin/main 5f5e4957, headless sway at 106x35 and 14 pt, load 2-14 with the CI runner active. The load for every figure is in the scratch logs. **The Linux vtebench scrolling rows are bound by where the scheduler places threads, not by felis's pipeline.** - **A bare PTY drain spans 77-260 ms.** The drain is a forkpty that reads back to back with no parsing, and its time depends on where the writer, the reader and the unbound `kworker` running `flush_to_ldisc` land: - flush worker on the writer's SMT sibling: 77-89 ms - all three on separate cores: about 130-150 ms - reader on the writer's sibling: 178-260 ms - **The placement is chance.** The worker stays on its previous CPU while that CPU is idle, and otherwise prefers a fully idle core. - **felis is fastest when the placement is favorable.** With the terminal pinned to CPUs 0-7 and vtebench to CPU 10, felis scored 82-103 ms, with the worker next to the writer 91-94% of the time. foot scored 93-206 ms in the same setup, with the worker next to the writer 51% of the time. Unpinned, felis scored 111-136 ms and foot 94-111 ms. - **Some gap remains at equal placement.** In two rounds where the placement was about the same, foot scored 104-107 ms and felis 120-128 ms. felis's reader makes about 1.2 M reads/s at about 10 B each, while foot's main thread makes about 0.4 M reads/s at about 38 B each. - **Levers tried without a clear gain:** - a 50 µs read spin - backing the parser off 1-4 ms while output streams - a 3 µs busy pause after short reads: about 8% better, within noise, and it moves kernel time into user-space spinning A reader that sleeps between reads still slows the producer. **Next steps:** - Confirm the mechanism with a kernel profile (`perf c2c`, or `perf record -a -g` on the writer and the flush worker in the three layouts). This needs `perf_event_paranoid` ≤ 1. - In the benchmark methodology, treat these rows as placement-bound: publish from many interleaved rounds, or add a pinned-layout variant.
natsukium commented 2026-09-25 20:57:37 +09:00
Author
Owner
Copy link

Parked for now. When this resumes, the next step is the kernel profile. Run perf c2c or perf record -a -g on the writer and the tty flush worker in the three layouts, to confirm the scheduling mechanism before changing felis or the benchmark method. This needs perf_event_paranoid ≤ 1.

Parked for now. When this resumes, the next step is the kernel profile. Run `perf c2c` or `perf record -a -g` on the writer and the tty flush worker in the three layouts, to confirm the scheduling mechanism before changing felis or the benchmark method. This needs `perf_event_paranoid` ≤ 1.
Commenting is not possible because the repository is archived.
No Branch/Tag specified
Branches Tags
main
v0.1.0-rc.9
v0.1.0-rc.8
v0.1.0-rc.7
v0.1.0-rc.6
v0.1.0-rc.5
v0.1.0-rc.4
v0.1.0-rc.3
v0.1.0-rc.2
v0.1.0-rc.1
Labels
Clear labels   
priority/P0
Must be resolved before the v0.1.0 tag

  
priority/P1
Must be settled before the first public contract freezes

  
priority/P2
Explicitly deferred until after v0.1.0

  
release/v0.1.0
Work tracked for the first tagged release

  
status/blocked
Cannot start now: waits on an upstream release, another issue, or a recorded revisit trigger

  
status/planned
Triaged with a source-grounded plan and a passed pi review; ready to implement

  
type/bug
Shipped behavior that contradicts the docs or the code's own intent; needs a product code change

  
type/design
An open scope or trade-off question; needs a recorded decision in the owning explanation doc before code

  
type/test-gap
Documented or reviewed behavior that no test pins; usually split from a review finding, needs no product code change

  
type/tracker
Parent issue that tracks child work

No labels
priority/P0
priority/P1
priority/P2
release/v0.1.0
status/blocked
status/planned
type/bug
type/design
type/test-gap
type/tracker
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
natsukium/felis#400
No description provided.