docs: reorganize cross-terminal benchmark docs and separate from testing #384

Merged

natsukium merged 2 commits from docs/organize-benchmarks into main 2026-09-25 00:17:57 +09:00

Conversation 0 Commits 2 Files changed 6 +295 -256

natsukium commented 2026-09-25 00:15:24 +09:00

Owner

Copy link

Summary

Consolidate all cross-terminal benchmark specifications and results into docs/reference/benchmarks.md, and separate the benchmark-harness architectural decisions from docs/explanation/testing.md into a dedicated docs/explanation/benchmarks.md.

Motivation & Rationale

1. Separation of Concerns between Testing and Benchmarking:
   docs/reference/testing.md and docs/explanation/testing.md govern felis's testing strategy, verification layers, and in-tree Criterion regression gating. Cross-terminal benchmarks compare felis against external peers (ghostty, kitty, alacritty, foot, wezterm) and do not act as CI regression gates. Keeping crossterm harness references and design decisions inside testing.md conflated verification with external performance benchmarking.

2. Single Source of Truth for Benchmark Reference:
   The cross-terminal harness specification (suites, metrics, window pinning, sampling, rounds) was previously documented in docs/reference/testing.md, while the raw measurement tables lived in docs/reference/benchmarks.md. All benchmark reference material now lives in docs/reference/benchmarks.md. In addition, historical commit and PR narrative descriptions were pruned from the result notes.

3. Dedicated Explanation for Benchmark Harness:
   The ~120 lines of architectural decisions behind the harness (why it is a local report rather than a CI gate, why repetition/rotation is required, workload choices and patches, latency instrument trade-offs between Typometer and wl-latency, first-frame startup measurement, and time-series sampling) represent substantial design decisions with clear trade-offs and rejection criteria. These now live in docs/explanation/benchmarks.md as an Explanation counterpart following Diátaxis principles.

Changes

• docs/reference/benchmarks.md:
  • Added the cross-terminal benchmark harness specification (suites, metrics, environment variables, invocation examples).
  • Cleaned up historical commit/PR narratives from macOS and Linux results sections.
• docs/reference/testing.md:
  • Removed cross-terminal benchmark sections and environment variables.
  • Linked to docs/reference/benchmarks.md where relevant.
• docs/explanation/benchmarks.md:
  • Created a dedicated page holding the harness design rationale, alternative rejections, and methodology.
• docs/explanation/testing.md:
  • Removed benchmark-harness decisions, keeping focus purely on testing strategy and in-tree Criterion regression gating.
• docs/README.md:
  • Updated the document map and table of contents to include docs/explanation/benchmarks.md.

## Summary Consolidate all cross-terminal benchmark specifications and results into `docs/reference/benchmarks.md`, and separate the benchmark-harness architectural decisions from `docs/explanation/testing.md` into a dedicated `docs/explanation/benchmarks.md`. ## Motivation & Rationale 1. **Separation of Concerns between Testing and Benchmarking**: `docs/reference/testing.md` and `docs/explanation/testing.md` govern felis's testing strategy, verification layers, and in-tree Criterion regression gating. Cross-terminal benchmarks compare felis against external peers (ghostty, kitty, alacritty, foot, wezterm) and do not act as CI regression gates. Keeping crossterm harness references and design decisions inside `testing.md` conflated verification with external performance benchmarking. 2. **Single Source of Truth for Benchmark Reference**: The cross-terminal harness specification (suites, metrics, window pinning, sampling, rounds) was previously documented in `docs/reference/testing.md`, while the raw measurement tables lived in `docs/reference/benchmarks.md`. All benchmark reference material now lives in `docs/reference/benchmarks.md`. In addition, historical commit and PR narrative descriptions were pruned from the result notes. 3. **Dedicated Explanation for Benchmark Harness**: The ~120 lines of architectural decisions behind the harness (why it is a local report rather than a CI gate, why repetition/rotation is required, workload choices and patches, latency instrument trade-offs between Typometer and wl-latency, first-frame startup measurement, and time-series sampling) represent substantial design decisions with clear trade-offs and rejection criteria. These now live in `docs/explanation/benchmarks.md` as an Explanation counterpart following Diátaxis principles. ## Changes - `docs/reference/benchmarks.md`: - Added the cross-terminal benchmark harness specification (suites, metrics, environment variables, invocation examples). - Cleaned up historical commit/PR narratives from macOS and Linux results sections. - `docs/reference/testing.md`: - Removed cross-terminal benchmark sections and environment variables. - Linked to `docs/reference/benchmarks.md` where relevant. - `docs/explanation/benchmarks.md`: - Created a dedicated page holding the harness design rationale, alternative rejections, and methodology. - `docs/explanation/testing.md`: - Removed benchmark-harness decisions, keeping focus purely on testing strategy and in-tree Criterion regression gating. - `docs/README.md`: - Updated the document map and table of contents to include `docs/explanation/benchmarks.md`.

natsukium added 2 commits 2026-09-25 00:15:25 +09:00

docs: consolidate cross-terminal benchmark reference into benchmarks.md ... c61a27ac49

Cross-terminal comparison harness details (suite table, memory
accounting, resource sampling, window pins, and measurement procedures)
previously lived in reference/testing.md, forcing readers of
reference/benchmarks.md to jump between files to interpret published
figures, while cluttering reference/testing.md with external comparison
specifications.

Moving the crossterm specification into reference/benchmarks.md
establishes a single reference owner for cross-terminal benchmarking
(both methodology and published results), while keeping
reference/testing.md focused on in-tree automated test layers (including
Criterion regression gates).

Additionally, stripped commit and PR history narratives from the macOS
and Linux result sections of benchmarks.md per doc-cascade rules.
Updated reference links across docs/README.md,
docs/explanation/testing.md, and skills to point to the consolidated
owner.

docs: separate cross-terminal benchmarks from testing strategy ... fe6e32b3a7

Testing reference and explanation (reference/testing.md and
explanation/testing.md) exist to verify felis's correctness, invariants,
and regression prevention. Comparative benchmarking against other
terminals (tools/bench/crossterm.py) is not a test or regression gate,
and its inclusion in the testing documents blurred the boundary between
in-tree verification and peer comparisons.

This removes the remaining crossterm cross-reference and environment
variables from reference/testing.md, and moves the cross-terminal
harness decisions out of explanation/testing.md into a dedicated
docs/explanation/benchmarks.md page.

This establishes two clean, orthogonal Diátaxis pairs:
- Testing: reference/testing.md <-> explanation/testing.md
- Benchmarks: reference/benchmarks.md <-> explanation/benchmarks.md

natsukium merged commit ef96bb46de into main 2026-09-25 00:17:57 +09:00

natsukium deleted branch docs/organize-benchmarks 2026-09-25 00:17:57 +09:00

natsukium referenced this pull request from a commit 2026-09-25 00:17:58 +09:00

docs: reorganize cross-terminal benchmark docs and separate from testing (#384)

natsukium referenced this pull request 2026-09-25 00:48:41 +09:00

docs: fix inconsistencies found by a tree-wide docs and skills audit #395

natsukium referenced this pull request from a commit 2026-09-25 00:54:55 +09:00

docs: fix inconsistencies found by a tree-wide docs and skills audit (#395)

Commenting is not possible because the repository is archived.

Reviewers

No reviewers

Labels

Clear labels   

priority/P0

Must be resolved before the v0.1.0 tag

  

priority/P1

Must be settled before the first public contract freezes

  

priority/P2

Explicitly deferred until after v0.1.0

  

release/v0.1.0

Work tracked for the first tagged release

  

status/blocked

Cannot start now: waits on an upstream release, another issue, or a recorded revisit trigger

  

status/planned

Triaged with a source-grounded plan and a passed pi review; ready to implement

  

type/bug

Shipped behavior that contradicts the docs or the code's own intent; needs a product code change

  

type/design

An open scope or trade-off question; needs a recorded decision in the owning explanation doc before code

  

type/test-gap

Documented or reviewed behavior that no test pins; usually split from a review finding, needs no product code change

  

type/tracker

Parent issue that tracks child work

No labels

priority/P0

priority/P1

priority/P2

release/v0.1.0

status/blocked

status/planned

type/bug

type/design

type/test-gap

type/tracker

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

natsukium/felis!384

No description provided.