docs: publish the Linux cross-terminal benchmark results #374

Merged

natsukium merged 12 commits from docs/linux-benchmarks into main 2026-09-24 12:02:04 +09:00

Conversation 1 Commits 12 Files changed 35 +2158 -351

natsukium commented 2026-09-24 01:15:58 +09:00

Owner

Copy link

Summary

Adds a Linux run to docs/reference/benchmarks.md next to the existing macOS run: three rounds on an Intel
i5-12400F + RTX 3080 under niri, against kitty 0.49.0 (nixpkgs master), alacritty, wezterm, ghostty tip and foot.
The macOS content moves under its own ## macOS heading, and each platform's charts get their own directory.

Two harness fixes found while measuring:

• bench: key chart colors by terminal — a chart missing one terminal shifted every hue, so kitty was drawn in
  felis's blue.
• bench: require the niri pin to hold for a second — ghostty tip applied a stale resize after the pin reported
  success, and three of its legs ran at 106x7.

Open question before merging

felis's window stops repainting under sustained output on this machine. Run alone under DOOM-fire it showed no
frame for 16 s while DOOM-fire reported 1,680 fps, and it stalled partway through termbench-pro. The page drops
felis's DOOM-fire result and states that its Linux throughput figures are upper bounds. Either merge with that caveat,
or fix the renderer first and re-measure felis.

Left out of the tables

• ghostty tip latency: failed all three rounds (glyphs arrive late; completed short runs measured 42–66 ms mean).
• foot launch time: with true as its child, foot exits without mapping a window, so hyperfine times no window.
• Rows marked "median of 2 rounds": three ghostty-tip legs at 106x7 and one failed alacritty latency round were
  discarded. The niri.py fix is not yet exercised by a live run.

Checks

• python3 tools/bench/crossterm_test.py: 204 tests pass.
• just docs-links, prose check, treefmt: pass.

## Summary Adds a Linux run to `docs/reference/benchmarks.md` next to the existing macOS run: three rounds on an Intel i5-12400F + RTX 3080 under niri, against kitty 0.49.0 (nixpkgs master), alacritty, wezterm, ghostty tip and foot. The macOS content moves under its own `## macOS` heading, and each platform's charts get their own directory. Two harness fixes found while measuring: - `bench: key chart colors by terminal` — a chart missing one terminal shifted every hue, so kitty was drawn in felis's blue. - `bench: require the niri pin to hold for a second` — ghostty tip applied a stale resize after the pin reported success, and three of its legs ran at 106x7. ## Open question before merging **felis's window stops repainting under sustained output on this machine.** Run alone under DOOM-fire it showed no frame for 16 s while DOOM-fire reported 1,680 fps, and it stalled partway through termbench-pro. The page drops felis's DOOM-fire result and states that its Linux throughput figures are upper bounds. Either merge with that caveat, or fix the renderer first and re-measure felis. ## Left out of the tables - ghostty tip latency: failed all three rounds (glyphs arrive late; completed short runs measured 42–66 ms mean). - foot launch time: with `true` as its child, foot exits without mapping a window, so hyperfine times no window. - Rows marked "median of 2 rounds": three ghostty-tip legs at 106x7 and one failed alacritty latency round were discarded. The `niri.py` fix is not yet exercised by a live run. ## Checks - `python3 tools/bench/crossterm_test.py`: 204 tests pass. - `just docs-links`, prose check, treefmt: pass.

natsukium added 3 commits 2026-09-24 01:15:58 +09:00

bench: key chart colors by terminal rather than by bar position ... 13540b75b8

report.py promises one hue per terminal across every chart, but it handed
out color slots in the order the bars were drawn. A chart that leaves one
terminal out (felis's DOOM-fire result dropped from the Linux run, or a
suite a terminal failed) shifted every later terminal down a slot, so
kitty was drawn in felis's blue and a reader carrying colors between
charts compared the wrong bars. Slots now come from each terminal's
position in TERM_ORDER, which stays fixed whoever is missing.

Assisted-by: Claude Code:Opus 5.5

bench: require the niri pin to hold for a second before releasing a leg ... 7cf4685f35

ghostty tip on Linux applies the configures of an earlier resize attempt
after a later attempt has already read back as the pinned grid. The pin
reported success, the wrapper recorded its final size a moment later, and
the workload ran at 106x7 instead of 106x35: three of eighteen ghostty-tip
legs in one three-round run, each discarded afterwards. Re-reading the
tty size after one second and only then accepting it catches the late
resize and sends the loop around again.

Assisted-by: Claude Code:Opus 5.5

docs: publish the Linux cross-terminal benchmark results ... 7c8e19f2c3

The benchmarks page held one macOS run, so nothing said how felis fares on
the platform most of its users run. This adds a three-round run on an
Intel i5-12400F with an RTX 3080 under niri, against kitty 0.49.0 (nixpkgs
master), alacritty, wezterm, ghostty tip and foot, and nests the macOS run
under its own heading, with each platform's charts in its own directory.
The two sections are not comparable with each other (hardware, pinned
font, display pipeline and latency instrument all differ), and the page
says so.

The CI runner and the Windows CI guest that share the machine were
stopped for the run: an earlier attempt with them up halved kitty's
DOOM-fire rate between rounds.

What the page leaves out, and why:

- felis's window stopped repainting under sustained output. Run alone
  under DOOM-fire it showed no frame for 16 s while DOOM-fire reported
  1,680 fps, and it stalled partway through termbench-pro. Its DOOM-fire
  result is dropped and its throughput figures are marked as upper
  bounds until the cause is found.
- ghostty tip's latency failed in every round (glyphs arrive after the
  instrument has moved to the next cell; completed shorter runs measured
  42 to 66 ms mean).
- foot's launch time: with `true` as its child, foot exits without
  mapping a window (niri reports no window for it), so hyperfine times no
  window. Every other terminal was checked to map one.
- Three ghostty-tip legs that ran at 106x7 and one alacritty latency round
  that failed; those rows carry two of three rounds.

The root README's feature bullet quoted the macOS figures without naming
the machine; on Linux felis idles at 65 MB, not under 33 MB, so the bullet
now names the M4 Max.

Assisted-by: Claude Code:Opus 5.5

natsukium referenced this pull request 2026-09-24 01:16:53 +09:00

render: the window stops repainting under sustained output on Linux #375

natsukium commented 2026-09-24 01:16:59 +09:00

Author

Owner

Copy link

Tracked in #375: the page drops felis's Linux DOOM-fire result and marks its throughput figures as upper bounds until that is fixed.

Tracked in #375: the page drops felis's Linux DOOM-fire result and marks its throughput figures as upper bounds until that is fixed.

natsukium referenced this pull request 2026-09-24 03:09:53 +09:00

ci: discard earlier Criterion results before the regression gate benches #377

natsukium added 7 commits 2026-09-24 03:39:07 +09:00

bench: read macOS CPU and footprint through one rusage v4 call ... 330dca2ec5

The memory suite's macOS CPU column came from `ps -o cputime`: 10 ms
resolution and one fork per pid, which is too coarse for a flood that
costs a fast terminal a fraction of a second and too slow for a sampler
that reads a leg four times a second. It now comes from
`proc_pid_rusage(RUSAGE_INFO_V4)`, whose `ri_user_time` and
`ri_system_time` are Mach absolute-time ticks and are converted through
`mach_timebase_info` (125/3 ns per tick on Apple Silicon; read as
nanoseconds they undercount by a factor of about 42).

The footprint comes from the same v4 struct. Its only peak is the
lifetime maximum, which on macOS is the 280-440 MB every GUI terminal
touches while it starts, so macOS still reports no peak rather than a
number unrelated to the workload. The docstring no longer claims vmmap
cannot read a hardened-runtime process: it can (footprint and vmmap both
read ghostty tip); the reason not to fork it is cost.

`child_pids` matched a substring of `ps -o comm=`, which macOS prints as
a full path, so kitty's `kitten __atexit__` helper counted only because
the install path contains "kitty", and any process installed under a
directory named after a terminal would have counted too. It now matches
the executable's basename, with kitty's `kitten` named explicitly as a
helper that belongs to kitty's footprint.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

bench: let wl-latency wait for screen states instead of reading them once ... b6376d7d43

ghostty tip failed every Linux latency round ("no row carries 5 evenly
spaced marks"). Driven by hand under niri on an RTX 3080, the cause is its
presentation, not its input: with the reference pattern written straight
to the pty (no keyboard involved) the first capture that holds all five
glyphs arrives anywhere from 30 ms to 1.85 s later. Frames captured in
between are partly drawn (glyphs and the bar cursor with rows missing) and
are replaced only on redraws about 600 ms apart. ghostty 1.3.1 and kitty
show the whole pattern within 40 ms under the same test, and ghostty tip
with GSK_RENDERER=cairo does too, so it is the GL path of that build.

Three instrument steps read the screen once, at a fixed moment, and so
turned that slow presentation into a failure rather than a number:

- Pattern detection captured once, 300 ms after typing (Typometer's
  constant). It now polls from 300 ms until the pattern is located or
  ANSWER_TIMEOUT runs out, and logs how long it took.
- The key was held until its glyph showed, as upstream does. A Wayland
  client repeats a held key itself, so a glyph slower than the seat's
  repeat delay typed a second period into the next cell, which the next
  sample then read as a block cursor. The key is now released right after
  the press; terminals act on the press, and the other five terminals
  measure the same with either (alacritty 4.3-6.0 ms mean, foot 2.9).
- A cell that is not background before its key used to abort as a block
  cursor. It is now waited out up to ANSWER_TIMEOUT and counted
  (`covered_before_key` in the JSON); a real block cursor never clears
  and still fails the leg.

With these, ghostty tip completes 100-key legs at 190-315 ms mean and a
maximum of 1.81 s, which is what its screen shows.

Two further changes target the one alacritty round that failed with "the
line would not clear" and that 40 relaunches (including right after a
closing kitty window, the order that round ran in) did not reproduce.
That round located its pattern one pixel row lower than every other
alacritty leg, which is what a window still settling from its pin would
produce. The "before" capture is now taken only once two captures 100 ms
apart agree, and a deletion that does not read back resends the
backspaces once (an erase at the start of a canonical-mode line is
ignored, so this only recovers a lost key). A deletion that still fails
now names the pixel it read, the background it expected, and whether
niri moved the window since it was located.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

bench: give a failed latency leg one more window, and quote why ... ff100e5ca3

wl-latency now waits out a slow frame instead of failing on it, but a
transient can still fail a leg: one alacritty round of the Linux run
reported "the line would not clear" and was never reproduced in 40
relaunches. A leg the instrument failed is now measured once more in a
fresh window. A terminal that cannot be measured fails both attempts; a
leg that was skipped (no focus, no quiet desktop) is not retried, since a
second window would be skipped for the same reason.

The retry must not make a retried bar look like a clean one, so the
instrument's last stderr line is appended to `<name>.latency-failed` and
the latency chart quotes every failed attempt and whether the leg was
measured after all. The chart also names the legs where a cell was found
already painted before its key and cleared on its own (the instrument's
`covered_before_key`): that is a frame the terminal presented and then
replaced, which the waiting would otherwise make invisible.

`crossterm.py run --terminals` limits the field, as `suites.py` already
could, so a multi-round run of one suite can be pointed at the legs under
investigation without the rest of the field.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

bench: time startup to the first window instead of to the process exit ... dc015bead0

The startup suite timed `<terminal> -e true` with hyperfine, which is a
process lifetime, and for part of the field no window was ever involved:
foot on Linux exits in 18 ms without mapping a window, wezterm on macOS
creates one it never orders on screen (111 ms), and felis maps one it
never presents a frame into when the child is `true`. The bars compared
teardown paths, not how long a user waits for a terminal.

Each launch now runs `sleep` so the window has a reason to exist, is
timed until the platform reports the launch's first window, and is then
terminated: under niri, the WindowOpenedOrChanged event for a window whose
pid is in the launch's process tree (a toplevel is mapped only once its
first buffer is committed, so this is time to first frame); on macOS, the
first on-screen layer-0 window of the tree in CGWindowListCopyWindowInfo,
polled every 4 ms, which proves the window is visible but not that it has
been drawn into, and the chart says so. A launch that shows no window
within 20 s is recorded as that failure and the leg gets no number. felis
keeps its cold (daemon killed before each launch) and warm rows; the warm
daemon's sessions are cleared between launches so each one finds the
same roster.

The lifetime number is dropped rather than kept beside it: where a window
maps it adds only the teardown, which no user waits on, and where none
maps it is the misleading number this replaces. Older results roots still
chart, under a subtitle naming them process lifetimes.

Measured with RUNS=5 WARMUP=2 (medians): Linux/niri foot 26, alacritty
113, wezterm 127, kitty 162, ghostty 196, ghostty tip 198, felis warm 171
and cold 250 ms; macOS alacritty 219, wezterm 198, ghostty tip 213 (both
unmeasurable before), kitty 318, felis cold 131 and warm 145-165 ms. On
macOS felis's warm launch is slower than its cold one in every pass, with
or without the session clearing, which the lifetime number hid.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

bench: record what each terminal spends on the throughput workloads ... b0fbdbe5d2

The throughput suites said how fast a terminal drains a workload and
nothing about what that cost it, although ghostty tip on Linux turned out
to spend 3-6 cores on workloads the rest of the field handles on one.
vtebench, termbench, cat, kittenbench and doom-fire now write
`<name>.res.json` per leg: CPU seconds (growth of utime + stime over the
window, with the wall time beside it for a share) and peak and mean
memory under the memory suite's ledger (PSS on Linux, phys_footprint on
macOS), sampled every 0.25 s. The report adds one CPU chart and one
peak-memory chart with a panel per suite, and notes each terminal's CPU
share and felis's client/daemon split.

The window is a handshake: the wrapper touches `.start` and waits for
`.started`, which the sampler writes after resolving the process set and
taking its baseline. With a marker the sampler merely noticed, resolving
the processes (a `ps` fork per child on macOS) outlasted a 300k-line
`seq`, and every terminal charted about zero CPU. After `.end` the
window stays open until the processes go idle, at most 3 s, because
`cat` returns once its bytes are in the pty and an eagerly reading
terminal is still parsing them.

The peak is the largest sample, not a kernel high-water mark: on macOS
the only one is the lifetime maximum, which for every GUI terminal is its
280-440 MB startup. CPU is what the terminal's own processes are charged,
so a GPU driver's threads and the compositor are outside it, and the
chart says so. Where sampling fails, the leg keeps its throughput number
and prints why.

The memory suite's CPU chart now names the children it counts (those
named after the terminal or a listed helper such as kitty's kitten), as
`child_pids` matches them.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

docs: describe the startup, resource and latency harness as it now runs ... d2ad2bcd0f

The testing reference described startup as a launch of `true` timed to
its exit, and said nothing about the per-leg resource records or how
wl-latency now waits for screen states and retries a failed leg; readers
reproducing a run would have looked for artifacts that no longer exist
and missed the ones that do. The explanation page now records why
startup ends at the first window (the lifetime alternative times a
teardown for part of the field), why resource peaks are sampled rather
than read from a kernel high-water mark, and the two places the latency
loop leaves Typometer's step-for-step shape.

The published Linux page named the wrong cause for ghostty tip's failed
latency legs; it now states the one found by driving the build by hand:
partly drawn frames completed only on redraws about 600 ms apart, read
by an instrument that looked once, 300 ms after typing.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

skills: point perf-trace's harness notes at the first-window startup and sampler ... a2997aa652

The notes still said the startup suite probed every command under a 30 s
guard because ghostty never exits after an instant command, and listed
only hyperfine's JSON among the chart inputs. An agent following them
would look for the wrong artifacts and misread why launches are
terminated. They now name `*.startup.json` and `*.res.json` and say that
every suite ends its own launches.

Assisted-by: Claude Code:Opus 5.5
Assisted-by: Claude Code

natsukium force-pushed docs/linux-benchmarks from a2997aa652 to 75d0863035 2026-09-24 09:23:24 +09:00 Compare

natsukium added 2 commits 2026-09-24 11:48:46 +09:00

bench: give each latency sample a one-second round-trip allowance ... 8ca4f0bfdd

The leg timeout allowed 60 ms of round trip per sample on top of the
typing delay. ghostty tip on Linux averages close to a second per key
under the harness, so every one of its latency legs timed out and the
suite recorded no result for it. The round trip is the quantity being
measured, so the budget has to cover a slow terminal; a fast one still
finishes early and pays nothing for the slack.

Assisted-by: Claude Code

docs: republish the Linux benchmarks from a run in which every leg completed ... 27c5fb134d

The earlier Linux results were taken while felis's window froze under
sustained output, so its throughput figures were upper bounds and its
DOOM-fire result was missing; ghostty-tip's latency and foot's startup
were not measured at all. With the parse-lock hand-off in the daemon and
the reworked harness, every leg of the new run completed in all three
rounds, so the section now reports the full field, startup as time to
first window, and the CPU time and peak memory each workload cost.

Two caveats keep the page honest. felis's DOOM-fire count is not marked
best: the program counts frames written, and felis painted only about
15 per second during the flood (tracked separately). And PSS moved by
55 to 86 MB for every GPU terminal between two runs a day apart while
foot's barely moved, because shared driver pages are split among every
process on the desktop that maps them, so the memory figures compare
only within one run.

Assisted-by: Claude Code

natsukium merged commit 27c5fb134d into main 2026-09-24 12:02:04 +09:00

Commenting is not possible because the repository is archived.

Reviewers

No reviewers

Labels

Clear labels   

priority/P0

Must be resolved before the v0.1.0 tag

  

priority/P1

Must be settled before the first public contract freezes

  

priority/P2

Explicitly deferred until after v0.1.0

  

release/v0.1.0

Work tracked for the first tagged release

  

status/blocked

Cannot start now: waits on an upstream release, another issue, or a recorded revisit trigger

  

status/planned

Triaged with a source-grounded plan and a passed pi review; ready to implement

  

type/bug

Shipped behavior that contradicts the docs or the code's own intent; needs a product code change

  

type/design

An open scope or trade-off question; needs a recorded decision in the owning explanation doc before code

  

type/test-gap

Documented or reviewed behavior that no test pins; usually split from a review finding, needs no product code change

  

type/tracker

Parent issue that tracks child work

No labels

priority/P0

priority/P1

priority/P2

release/v0.1.0

status/blocked

status/planned

type/bug

type/design

type/test-gap

type/tracker

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

natsukium/felis!374

No description provided.