bench: promote a leg's artifacts only from a completed attempt #433

Merged

natsukium merged 9 commits from bench/atomic-leg-results into main 2026-09-25 12:29:38 +09:00

Conversation 0 Commits 9 Files changed 7 +1017 -318

natsukium commented 2026-09-25 11:16:05 +09:00

Owner

Copy link

Closes #426

A leg's processes wrote straight into the suite directory the report reads, so keeping stale or late artifacts out of the report depended on cleanup at every failure site. This replaces that with one structural mechanism in tools/bench/field.py:

• Leg attempts. Each attempt at a leg writes everything (workload output, markers, the harness's own records) into a fresh .attempts/<name>.* directory. Starting an attempt deletes what earlier attempts of that leg left in the suite directory (all of felis's startup rows included), except the harness-appended <name>.latency-failed. At the end of the leg, before its processes are stopped, the attempt is copied into a sealed directory; later harness records go there. An attempt that returns is committed by renaming the sealed copy to .attempts/<name>.promoting and then moved into place, resume marker last; a crash mid-promotion is finished before the next suite and before the report reads. An attempt that raises or is interrupted is discarded; one that was refused, or that returned without its resume marker, lands only its env record and sets the rest aside in refused/ or unfinished/. This replaces sweep_refused.
• Stopping by identity. A leg's processes are held through the harness's own Popen, a pidfd on Linux, or a start time on macOS; the tree is walked before the first signal and on every poll while waiting, and SIGKILL goes only through those handles. Used by the leg teardown, felis's client/daemon stop, and the startup suite's launch and daemon stops (which previously SIGKILLed every pid it had read, including exited ones). A process the walk misses costs a LINGERING line, never a wrong bar.

Docs: docs/reference/benchmarks.md "Leg attempts" and the stop sequence; docs/explanation/benchmarks.md records why promotion replaced per-failure-site cleanup.

Verified: python3 tools/bench/crossterm_test.py (283 tests), and real windows on Linux/niri (crossterm.py run startup kittenbench memory --terminals kitty), including a harness killed mid-leg (nothing of that leg in the suite directory) and its resume (the leg retaken, .attempts/ cleared). No macOS run: the macOS branch (start-time identity) is covered by mocked tests only.

Closes #426 A leg's processes wrote straight into the suite directory the report reads, so keeping stale or late artifacts out of the report depended on cleanup at every failure site. This replaces that with one structural mechanism in `tools/bench/field.py`: - **Leg attempts.** Each attempt at a leg writes everything (workload output, markers, the harness's own records) into a fresh `.attempts/<name>.*` directory. Starting an attempt deletes what earlier attempts of that leg left in the suite directory (all of felis's startup rows included), except the harness-appended `<name>.latency-failed`. At the end of the leg, before its processes are stopped, the attempt is copied into a sealed directory; later harness records go there. An attempt that returns is committed by renaming the sealed copy to `.attempts/<name>.promoting` and then moved into place, resume marker last; a crash mid-promotion is finished before the next suite and before the report reads. An attempt that raises or is interrupted is discarded; one that was refused, or that returned without its resume marker, lands only its env record and sets the rest aside in `refused/` or `unfinished/`. This replaces `sweep_refused`. - **Stopping by identity.** A leg's processes are held through the harness's own `Popen`, a pidfd on Linux, or a start time on macOS; the tree is walked before the first signal and on every poll while waiting, and SIGKILL goes only through those handles. Used by the leg teardown, felis's client/daemon stop, and the startup suite's launch and daemon stops (which previously SIGKILLed every pid it had read, including exited ones). A process the walk misses costs a `LINGERING` line, never a wrong bar. Docs: `docs/reference/benchmarks.md` "Leg attempts" and the stop sequence; `docs/explanation/benchmarks.md` records why promotion replaced per-failure-site cleanup. Verified: `python3 tools/bench/crossterm_test.py` (283 tests), and real windows on Linux/niri (`crossterm.py run startup kittenbench memory --terminals kitty`), including a harness killed mid-leg (nothing of that leg in the suite directory) and its resume (the leg retaken, `.attempts/` cleared). No macOS run: the macOS branch (start-time identity) is covered by mocked tests only.

natsukium added 6 commits 2026-09-25 11:16:05 +09:00

bench: promote a leg's artifacts only from a completed attempt ... c38aa02324

A leg's processes wrote straight into the suite directory the report
reads, so what reached the report depended on cleanup at every failure
site: a retry that raised before rewriting its rows left an interrupted
attempt's row charted, a launch that ignored SIGTERM or a child forked
after the tree was read could still write a result or resume marker
after the leg was stopped, and felis's wrapper runs under the daemon,
outside the launch's tree altogether.

Each attempt now writes everything, the workload's output and the
harness's own records, into a fresh directory under .attempts/. Starting
an attempt deletes what earlier attempts of that leg left behind; an
attempt that returns is copied into a staging directory, committed by
one rename, and moved into place with the resume marker last, and one
that raises is discarded. The copy is the point: a writer still holding
an attempt file open writes into the attempt's inode, never the promoted
one, and a path-based late write finds its directory gone. This
replaces sweeping refused legs before a suite and a report; a refused
attempt lands its env record and sends the rest to refused/.

Stopping a leg now holds every process by identity (the launch through
its Popen, others through a pidfd on Linux or a start time on macOS),
walks the tree before the first signal and while waiting, and escalates
to SIGKILL only through those handles. The startup suite's stop sent
SIGKILL to every pid it had read, including ones that had already
exited and could have been reused.

Assisted-by: Claude Code

docs: specify leg attempts and how a leg's processes are stopped ... 29c4b05ea6

The harness spec described sweeping a refused leg's artifacts before a suite and a report. Artifacts now reach the suite directory only from a completed attempt, and a leg's processes are stopped by identity, so the reference states the attempt layout, promotion and stop sequence, and the explanation records why promotion replaced cleanup at each failure site: that is the alternative a contributor would reach for, and the reference facts alone do not show which windows it leaves open.

Assisted-by: Claude Code

bench: skip already-held processes when re-walking a leg's tree ... b934b53875

The stop re-walks the tree on every poll while it waits, and holding a child costs a ps call to confirm its parent. Re-holding every live child each time only to discard the duplicate spent that fork per process per poll; a pid that belongs to a live held process is that process, so it is skipped before it is held.

Assisted-by: Claude Code

bench: keep a refused promotion refused when it is finished after a crash ... 59eea72be6

Promotion read the refusal from the staged env record, which is the first file moved into the suite directory. A promotion cut short after that move was finished as an ordinary one, landing the refused attempt's result and resume marker where the report and the resume check read them. The staging now carries its own refusal mark, which stays until the staged directory is removed.

Assisted-by: Claude Code

bench: set aside an attempt that returned without its resume marker ... efca4cf3e0

A leg that timed out or whose driver failed still returned normally, so its attempt was promoted whole, and a number a workload wrote before it stopped (an instrument that wrote its JSON and then exited nonzero, a benchmark cut off after some categories) was charted beside the finished legs. Only an attempt that reached its resume marker now lands its files; any other lands its env record, which the report's notes read, and sets the rest aside in unfinished/ the way a refused attempt uses refused/. The memory suite has no .done, so its record is the marker.

Assisted-by: Claude Code

bench: seal a leg's output before stopping its processes ... 3e4bf1abb1

Promotion copied the attempt directory after the leg's processes were stopped, so a process the stop never reached could still rewrite a result between the stop and the copy, and a leg that already had its resume marker would land that write. The attempt is now copied into a sealed directory at the end of the leg, before the stop; later harness records of the attempt go to the sealed copy, and it is the sealed copy that is committed and promoted. Nothing written into the attempt directory after that point, through a path or an open file, reaches the report.

Assisted-by: Claude Code

natsukium added 2 commits 2026-09-25 11:59:47 +09:00

bench: read a leg's process tree from /proc on Linux ... 138bd33a5c

Holding a child confirmed its parent through ps -o ppid= and listed children through pgrep -P. On the CI host the stop tests found the child they had forked still running after the stop reported nothing left, which is what happens when that ps output does not parse: the child is never held and so never signalled. /proc/<pid>/task/*/children and /proc/<pid>/stat are what those tools read anyway and need no particular procps on PATH; macOS keeps ps and pgrep. The forked-child test also now waits until its shell ignores SIGTERM before stopping it, since a stop that lands first kills the shell before it can fork the child the test is about.

Assisted-by: Claude Code

Merge remote-tracking branch 'origin/main' into bench/atomic-leg-results 9168b1258a

natsukium added 1 commit 2026-09-25 12:04:02 +09:00

bench: release the handshake test's leg on its resume marker ... f1ca4647eb

The test's workload touches .done after the file it copies, but the leg was released on that copy, so under load the leg could be sealed between the two and set aside as unfinished, which the test then read as a missing workload. The leg now waits for .done, the marker that decides whether it lands, with room for a loaded machine.

Assisted-by: Claude Code

natsukium merged commit 51b5d31ffc into main 2026-09-25 12:29:38 +09:00

natsukium deleted branch bench/atomic-leg-results 2026-09-25 12:29:38 +09:00

natsukium referenced this pull request from a commit 2026-09-25 12:29:39 +09:00

bench: promote a leg's artifacts only from a completed attempt (#433)

natsukium referenced this pull request 2026-09-25 13:15:57 +09:00

bench: follow up the 9d1d842f cross-terminal run #405

Commenting is not possible because the repository is archived.

Reviewers

No reviewers

Labels

Clear labels   

priority/P0

Must be resolved before the v0.1.0 tag

  

priority/P1

Must be settled before the first public contract freezes

  

priority/P2

Explicitly deferred until after v0.1.0

  

release/v0.1.0

Work tracked for the first tagged release

  

status/blocked

Cannot start now: waits on an upstream release, another issue, or a recorded revisit trigger

  

status/planned

Triaged with a source-grounded plan and a passed pi review; ready to implement

  

type/bug

Shipped behavior that contradicts the docs or the code's own intent; needs a product code change

  

type/design

An open scope or trade-off question; needs a recorded decision in the owning explanation doc before code

  

type/test-gap

Documented or reviewed behavior that no test pins; usually split from a review finding, needs no product code change

  

type/tracker

Parent issue that tracks child work

No labels

priority/P0

priority/P1

priority/P2

release/v0.1.0

status/blocked

status/planned

type/bug

type/design

type/test-gap

type/tracker

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

natsukium/felis!433

No description provided.