mutants: speed up cargo-mutants runs #273

New issue

Closed

opened 2026-09-13 11:46:17 +09:00 by natsukium · 1 comment

natsukium commented

2026-09-13 11:46:17 +09:00

Owner

Context

cargo mutants --list currently reports 5161 mutants under the existing scope (crates/felis-grid, crates/felis-vt, crates/felis-protocol). At that count, full runs are dominated by per-mutant incremental rebuild + test time: the last baseline took ~23s to build with an auto-set test timeout of 58s (see mutants.out/debug.log).

.cargo/mutants.toml is already in good shape: test_tool = "nextest", examine_globs restricted to the boundary-logic crates, and committed prost codegen plus cfg(kani) dead code excluded. No linker override and no [profile.mutants] exist yet.

Assessment of the commonly cited tweaks

Evaluated against this cargo-mutants performance post:

Fast linker (mold/wild): helps, with discounted expectations. Every mutant pays a relink, so shorter link time multiplies across 5161 mutants. But the mutants scope excludes the GPU/OS crates, so gains should be smaller than the "20% / half" figures quoted for link-heavy workspaces. Per repo policy the toolchain belongs to the Nix flake, so this must go through dev/flake-module.nix, not ad-hoc RUSTFLAGS.
Dedicated [profile.mutants] (inherits = "test", debug info off): helps, but inert on its own. Just adding the profile to Cargo.toml changes nothing until cargo-mutants selects it (--profile mutants / config profile). First run after adding it pays one full rebuild into a separate target dir.
Parallel -j: -j2 is too low here. That starting value is generic advice; on a 12-core / 62 GB machine we can go higher while watching memory (outer mutants -j multiplies with nextest's inner parallelism).
RAM disk TMPDIR: lowest priority, do not do first. /tmp here is disk-backed while /dev/shm (32G tmpfs) exists, so tmpfs scratch is possible — but it competes for memory with parallel builds. Only worth it if I/O proves to be the bottleneck.

Bigger wins than any single build tweak

Narrow the mutant population: --in-diff for PRs, --shard n/m for CI and local trials, -F/--file, -p, --iterate.
Tune timeouts: the auto test timeout was 58s, so slow suites (e.g. proptest-heavy packages) drag every mutant; consider --timeout-multiplier / minimum-test-timeout and #[mutants::skip] for hang-prone fns.
Do not add sccache for this: sources differ per mutant, so cache hits are negligible.

Proposed steps

Measure: cargo mutants --shard 0/8 for baseline build time and per-mutant average.
Add [profile.mutants] + select it via --profile mutants/config; compare on the same shard.
Raise -j while watching memory.
Add mold via the Nix dev shell; compare again.
Adopt --in-diff for PRs and --shard for full runs as the standard workflow.

Acceptance

Shard-measured before/after numbers recorded in the issue.
The chosen settings landed in .cargo/mutants.toml / Cargo.toml / Nix config as appropriate (no ad-hoc local-only flags).
Standard invocation for PR vs. full runs is repeatable via just mutants.

## Context `cargo mutants --list` currently reports **5161 mutants** under the existing scope (`crates/felis-grid`, `crates/felis-vt`, `crates/felis-protocol`). At that count, full runs are dominated by per-mutant incremental rebuild + test time: the last baseline took ~23s to build with an auto-set test timeout of 58s (see `mutants.out/debug.log`). `.cargo/mutants.toml` is already in good shape: `test_tool = "nextest"`, `examine_globs` restricted to the boundary-logic crates, and committed prost codegen plus `cfg(kani)` dead code excluded. No linker override and no `[profile.mutants]` exist yet. ## Assessment of the commonly cited tweaks Evaluated against [this cargo-mutants performance post](https://syu-m-5151.hatenablog.com/entry/2026/01/02/083735#%E3%83%91%E3%83%95%E3%82%A9%E3%83%BC%E3%83%9E%E3%83%B3%E3%82%B9%E6%9C%80%E9%81%A9%E5%8C%96): 1. **Fast linker (mold/wild): helps, with discounted expectations.** Every mutant pays a relink, so shorter link time multiplies across 5161 mutants. But the mutants scope excludes the GPU/OS crates, so gains should be smaller than the "20% / half" figures quoted for link-heavy workspaces. Per repo policy the toolchain belongs to the Nix flake, so this must go through `dev/flake-module.nix`, not ad-hoc `RUSTFLAGS`. 2. **Dedicated `[profile.mutants]` (`inherits = "test"`, debug info off): helps, but inert on its own.** Just adding the profile to `Cargo.toml` changes nothing until cargo-mutants selects it (`--profile mutants` / config `profile`). First run after adding it pays one full rebuild into a separate target dir. 3. **Parallel `-j`: `-j2` is too low here.** That starting value is generic advice; on a 12-core / 62 GB machine we can go higher while watching memory (outer mutants `-j` multiplies with nextest's inner parallelism). 4. **RAM disk `TMPDIR`: lowest priority, do not do first.** `/tmp` here is disk-backed while `/dev/shm` (32G tmpfs) exists, so tmpfs scratch is possible — but it competes for memory with parallel builds. Only worth it if I/O proves to be the bottleneck. ## Bigger wins than any single build tweak - Narrow the mutant population: `--in-diff` for PRs, `--shard n/m` for CI and local trials, `-F`/`--file`, `-p`, `--iterate`. - Tune timeouts: the auto test timeout was 58s, so slow suites (e.g. proptest-heavy packages) drag every mutant; consider `--timeout-multiplier` / `minimum-test-timeout` and `#[mutants::skip]` for hang-prone fns. - Do not add sccache for this: sources differ per mutant, so cache hits are negligible. ## Proposed steps 1. Measure: `cargo mutants --shard 0/8` for baseline build time and per-mutant average. 2. Add `[profile.mutants]` + select it via `--profile mutants`/config; compare on the same shard. 3. Raise `-j` while watching memory. 4. Add mold via the Nix dev shell; compare again. 5. Adopt `--in-diff` for PRs and `--shard` for full runs as the standard workflow. ## Acceptance - [ ] Shard-measured before/after numbers recorded in the issue. - [ ] The chosen settings landed in `.cargo/mutants.toml` / `Cargo.toml` / Nix config as appropriate (no ad-hoc local-only flags). - [ ] Standard invocation for PR vs. full runs is repeatable via `just mutants`.

natsukium referenced this issue

2026-09-13 14:23:05 +09:00

Speed up cargo-mutants runs #274

natsukium commented

2026-09-13 14:23:25 +09:00

Author

Owner

Measured and landed the effective tweaks in #274 (branch mutants-speed). All numbers below are same-scope
before/after on the 12-core / 62 GB dev machine.

1. `[profile.mutants]` (`inherits = "test"`, `debug = 0`) — helps, landed

Selected via profile = "mutants" in .cargo/mutants.toml (declaring it in Cargo.toml alone is inert).

Cold scratch build (mutants baseline): ~38s (test profile) -> ~16s (mutants profile)
Warm single-crate incremental rebuild (felis-grid touch + nextest --no-run): 3.05s -> 2.19s (~28%)
Full suite time unchanged: ~8.8s -> ~8.0s, same 1374 passed
5-mutant all-caught sweep at -j1: 62s -> 48s (~23%)

2. `-j` — `-j4` is the knee, landed as the `just mutants` default

18-mutant mixed set (-F 'PartialEq', 7 missed / 11 caught), --profile mutants:

-j	wall	notes
1	3m49s	baseline
2	2m32s	1.5x
4	2m26s	1.57x, best
8	3m18s	regresses: target-dir lock contention (85s+/94s stalled builds)

(On a 5-mutant all-caught micro-set -j1 beats -j4, 48s vs 64s — outer parallelism only pays once the run is
large enough to amortize contention, which every real sweep is.)

jobs cannot be pinned in .cargo/mutants.toml — the config schema rejects it (unknown field 'jobs') — so
the default lives in the just mutants* recipes via CARGO_MUTANTS_JOBS="${CARGO_MUTANTS_JOBS:-4}", still
overridable.

3. mold via the Nix dev shell — helps on top, landed

Warm single-crate rebuild under the mutants profile: 2.11s (BFD) -> 1.59s (mold), ~25% off the
link-dominated remainder; combined with the profile, 3.05s -> 1.59s (~48%). Verified end-to-end through a
mutants scratch run (env propagates to the child cargo invocations). Wired as Linux-only RUSTFLAGS in the
dev shell's shellHook, not .cargo/config.toml, so non-shell and macOS builds keep the default linker.

4. Standard workflow — landed

PRs: just mutants-pr (mutants touched by base...HEAD, default origin/main)
Full runs: just mutants-shard 0/8 (every shard), i.e. --shard for trials as proposed
Verified --in-diff takes a unified diff file path; -F remains the narrowing filter
(--file unions with examine_globs and widens instead)

Deliberately not done

RAM disk TMPDIR: the reflinked scratch copy is ~51 GB / ~93k files, larger than the 32 GB /dev/shm
tmpfs — it does not fit. Reflink copy on btrfs takes ~10s, not the bottleneck.
sccache: sources differ per mutant, cache hits negligible (as suspected).
Timeout tuning: the suite runs ~8s and the auto timeout settled at ~42s (was 58s); no evidence of a
timeout problem.
--copy-target / --gitignore tuning: the copied target/ dir warms the baseline build (18s vs 38s
cold), so skipping the copy to save ~10s would lose more than it gains.
--test-workspace false: kept at default per the skill's note — narrowing by workspace drops
cross-crate coverage and reports false survivors.

Acceptance

Shard-measured before/after numbers: above (same-scope -F sets; a full 0/8 shard was not re-run
end-to-end since each configuration point costs ~30 min — the -F sets isolate each variable).
Settings in .cargo/mutants.toml / Cargo.toml / Nix config, no ad-hoc local flags.
Repeatable via just mutants / just mutants-pr / just mutants-shard.

Measured and landed the effective tweaks in #274 (branch `mutants-speed`). All numbers below are same-scope before/after on the 12-core / 62 GB dev machine. ## 1. `[profile.mutants]` (`inherits = "test"`, `debug = 0`) — helps, landed Selected via `profile = "mutants"` in `.cargo/mutants.toml` (declaring it in `Cargo.toml` alone is inert). - Cold scratch build (mutants baseline): ~38s (test profile) -> ~16s (mutants profile) - Warm single-crate incremental rebuild (`felis-grid` touch + `nextest --no-run`): 3.05s -> 2.19s (~28%) - Full suite time unchanged: ~8.8s -> ~8.0s, same 1374 passed - 5-mutant all-caught sweep at -j1: 62s -> 48s (~23%) ## 2. `-j` — `-j4` is the knee, landed as the `just mutants` default 18-mutant mixed set (`-F 'PartialEq'`, 7 missed / 11 caught), `--profile mutants`: | -j | wall | notes | |----|------|-------| | 1 | 3m49s | baseline | | 2 | 2m32s | 1.5x | | 4 | 2m26s | 1.57x, best | | 8 | 3m18s | regresses: target-dir lock contention (85s+/94s stalled builds) | (On a 5-mutant all-caught micro-set -j1 beats -j4, 48s vs 64s — outer parallelism only pays once the run is large enough to amortize contention, which every real sweep is.) `jobs` cannot be pinned in `.cargo/mutants.toml` — the config schema rejects it (`unknown field 'jobs'`) — so the default lives in the `just mutants*` recipes via `CARGO_MUTANTS_JOBS="${CARGO_MUTANTS_JOBS:-4}"`, still overridable. ## 3. mold via the Nix dev shell — helps on top, landed Warm single-crate rebuild under the mutants profile: 2.11s (BFD) -> 1.59s (mold), ~25% off the link-dominated remainder; combined with the profile, 3.05s -> 1.59s (~48%). Verified end-to-end through a mutants scratch run (env propagates to the child cargo invocations). Wired as Linux-only `RUSTFLAGS` in the dev shell's `shellHook`, not `.cargo/config.toml`, so non-shell and macOS builds keep the default linker. ## 4. Standard workflow — landed - PRs: `just mutants-pr` (mutants touched by `base...HEAD`, default `origin/main`) - Full runs: `just mutants-shard 0/8` (every shard), i.e. `--shard` for trials as proposed - Verified `--in-diff` takes a unified diff file path; `-F` remains the narrowing filter (`--file` unions with `examine_globs` and widens instead) ## Deliberately not done - **RAM disk `TMPDIR`**: the reflinked scratch copy is ~51 GB / ~93k files, larger than the 32 GB `/dev/shm` tmpfs — it does not fit. Reflink copy on btrfs takes ~10s, not the bottleneck. - **sccache**: sources differ per mutant, cache hits negligible (as suspected). - **Timeout tuning**: the suite runs ~8s and the auto timeout settled at ~42s (was 58s); no evidence of a timeout problem. - **`--copy-target` / `--gitignore` tuning**: the copied `target/` dir warms the baseline build (18s vs 38s cold), so skipping the copy to save ~10s would lose more than it gains. - **`--test-workspace false`**: kept at default per the skill's note — narrowing by workspace drops cross-crate coverage and reports false survivors. ## Acceptance - [x] Shard-measured before/after numbers: above (same-scope `-F` sets; a full 0/8 shard was not re-run end-to-end since each configuration point costs ~30 min — the `-F` sets isolate each variable). - [x] Settings in `.cargo/mutants.toml` / `Cargo.toml` / Nix config, no ad-hoc local flags. - [x] Repeatable via `just mutants` / `just mutants-pr` / `just mutants-shard`.

natsukium referenced this issue from a commit

2026-09-13 14:57:26 +09:00

Speed up cargo-mutants runs (#274)

natsukium closed this issue

2026-09-13 14:57:37 +09:00