ci: discard earlier Criterion results before the regression gate benches #377

Merged

natsukium merged 1 commit from bench/gate-stale-results into main 2026-09-24 03:29:42 +09:00

Conversation 0 Commits 1 Files changed 1 +6

natsukium commented 2026-09-24 03:09:53 +09:00

Owner

Copy link

The Criterion regression gate failed on every PR, including docs-only #374 and the #375 fix #376. Both runs reported the same 44 regressions to the decimal.

Cause: criterion.py gate walks every change/estimates.json under $CARGO_TARGET_DIR/criterion, and that directory persists across jobs on the nix runner. The PR gate re-benches only daemon_side, multi_cycle and end_to_end_throughput, so change files left by earlier runs of other benches (socket_write, video_frame, parser_throughput, and more) were read as this PR's regressions. All 9 benches the job actually measured were within noise or faster.

Fix: remove the Criterion root before benching the base SHA.

Rejected alternative: filtering inside the gate would have to repeat the workflow's bench selection and could drift from it.

The Criterion regression gate failed on every PR, including docs-only #374 and the #375 fix #376. Both runs reported the same 44 regressions to the decimal. **Cause:** `criterion.py gate` walks every `change/estimates.json` under `$CARGO_TARGET_DIR/criterion`, and that directory persists across jobs on the nix runner. The PR gate re-benches only daemon_side, multi_cycle and end_to_end_throughput, so change files left by earlier runs of other benches (socket_write, video_frame, parser_throughput, and more) were read as this PR's regressions. All 9 benches the job actually measured were within noise or faster. **Fix:** remove the Criterion root before benching the base SHA. **Rejected alternative:** filtering inside the gate would have to repeat the workflow's bench selection and could drift from it.

natsukium added 1 commit 2026-09-24 03:09:53 +09:00

ci: discard earlier Criterion results before the regression gate benches ... b90ef0936e

The gate walks every change/estimates.json under the Criterion root, and
on the nix runner that root lives in a target directory that persists
across jobs. The PR gate only re-benches daemon_side, multi_cycle and
end_to_end_throughput, so change files left by an earlier run of other
benches were judged as the PR's regressions: every PR failed with the
same 44 stale results (socket_write, video_frame, parser_throughput, ...)
while the benches it actually measured were within noise or faster.

Clearing the root first keeps the gate to what this job measured. A
filter in the gate itself was rejected: it would have to repeat the
workflow's bench selection and drift from it.

Assisted-by: Claude Code

natsukium merged commit b90ef0936e into main 2026-09-24 03:29:42 +09:00

natsukium deleted branch bench/gate-stale-results 2026-09-24 03:29:42 +09:00

Commenting is not possible because the repository is archived.

Reviewers

No reviewers

Labels

Clear labels   

priority/P0

Must be resolved before the v0.1.0 tag

  

priority/P1

Must be settled before the first public contract freezes

  

priority/P2

Explicitly deferred until after v0.1.0

  

release/v0.1.0

Work tracked for the first tagged release

  

status/blocked

Cannot start now: waits on an upstream release, another issue, or a recorded revisit trigger

  

status/planned

Triaged with a source-grounded plan and a passed pi review; ready to implement

  

type/bug

Shipped behavior that contradicts the docs or the code's own intent; needs a product code change

  

type/design

An open scope or trade-off question; needs a recorded decision in the owning explanation doc before code

  

type/test-gap

Documented or reviewed behavior that no test pins; usually split from a review finding, needs no product code change

  

type/tracker

Parent issue that tracks child work

No labels

priority/P0

priority/P1

priority/P2

release/v0.1.0

status/blocked

status/planned

type/bug

type/design

type/test-gap

type/tracker

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

natsukium/felis!377

No description provided.