daemon: felis paints about 15 frames per second under a sustained flood #378

Closed

opened 2026-09-24 11:46:55 +09:00 by natsukium · 1 comment

natsukium commented

2026-09-24 11:46:55 +09:00

Owner

Follow-up to #375. With the ParseCore lock handed off (parking_lot::Mutex), felis no longer freezes under a flood, but it still paints far below the display rate while the flood runs.

Observed

Linux cross-terminal run of 2026-09-24 (i5-12400F, RTX 3080, niri, 60 Hz), DOOM-fire at a 141x61 grid:

The client sent 4 to 34 NextGridFrame pulls per second, about 15 on average. Each pull is one painted frame.
The DOOM-fire counter reported 2,618 fps for felis against 783 to 2,290 fps for the other five terminals.

The two numbers are linked: the fewer grid frames the daemon snapshots, the more of the lock the parse thread keeps, so felis's DOOM-fire figure counts parse work the other terminals spent partly on frames. The published figure carries this caveat until the paint rate is fixed.

Why

parking_lot's eventual fairness hands the lock to a waiter only about every 0.5 ms. Each pull needs a drain, a fan-out and a ship under that lock, and the parse thread wins every other contention, so a cycle waits several fairness slices.

Done when

felis paints at or near the display rate during DOOM-fire at the benchmark grid (a pull rate within reach of 60 per second on a 60 Hz output).
cat of the 150 MB payloads shows no regression.
The DOOM-fire caveat in docs/reference/benchmarks.md is removed or restated.

Candidates, not decided: bound the parse thread's hold per swap buffer, give the pull path an explicit hand-off (a fair unlock when a pull is pending), or snapshot outside the lock.

Follow-up to #375. With the `ParseCore` lock handed off (`parking_lot::Mutex`), felis no longer freezes under a flood, but it still paints far below the display rate while the flood runs. ## Observed Linux cross-terminal run of 2026-09-24 (i5-12400F, RTX 3080, niri, 60 Hz), DOOM-fire at a 141x61 grid: - The client sent 4 to 34 `NextGridFrame` pulls per second, about 15 on average. Each pull is one painted frame. - The DOOM-fire counter reported 2,618 fps for felis against 783 to 2,290 fps for the other five terminals. The two numbers are linked: the fewer grid frames the daemon snapshots, the more of the lock the parse thread keeps, so felis's DOOM-fire figure counts parse work the other terminals spent partly on frames. The published figure carries this caveat until the paint rate is fixed. ## Why `parking_lot`'s eventual fairness hands the lock to a waiter only about every 0.5 ms. Each pull needs a drain, a fan-out and a ship under that lock, and the parse thread wins every other contention, so a cycle waits several fairness slices. ## Done when - felis paints at or near the display rate during DOOM-fire at the benchmark grid (a pull rate within reach of 60 per second on a 60 Hz output). - `cat` of the 150 MB payloads shows no regression. - The DOOM-fire caveat in `docs/reference/benchmarks.md` is removed or restated. Candidates, not decided: bound the parse thread's hold per swap buffer, give the pull path an explicit hand-off (a fair unlock when a pull is pending), or snapshot outside the lock.

natsukium commented

2026-09-24 12:28:56 +09:00

Author

Owner

Traced the drain path in crates/felis-daemon/src/serve/session_task.rs: drain_and_fan takes and releases the ParseCore lock 5-6 separate times per drain cycle rather than once:

drain_effects (session_task.rs:1589, plus one more inside streaming::drain_effects)
fan_out_grid_state (session_task.rs:1665 and :1684, two acquisitions)
fan_out_notifications inside after_drive (session_task.rs:1631)
broadcast_facets (session_task.rs:1701)
ship_sub_at, once per subscriber (session_task.rs:1788)

parking_lot's eventual fairness hands the lock to a waiter only about once per 0.5 ms, while the parse thread relocks back-to-back between chunks. Completing one pull needs every one of these 5-6 acquisitions to independently win that fairness lottery before the parse thread reclaims the lock, which matches the observed 4-34 fps spread under a sustained flood.

Candidate fix: acquire the lock once per drain cycle and hold it across drain, fan-out and ship, instead of the current lock/unlock sequence. That cuts the number of required fairness wins from 5-6 to 1 per pull.

Traced the drain path in `crates/felis-daemon/src/serve/session_task.rs`: `drain_and_fan` takes and releases the `ParseCore` lock 5-6 separate times per drain cycle rather than once: 1. `drain_effects` (`session_task.rs:1589`, plus one more inside `streaming::drain_effects`) 2. `fan_out_grid_state` (`session_task.rs:1665` and `:1684`, two acquisitions) 3. `fan_out_notifications` inside `after_drive` (`session_task.rs:1631`) 4. `broadcast_facets` (`session_task.rs:1701`) 5. `ship_sub_at`, once per subscriber (`session_task.rs:1788`) `parking_lot`'s eventual fairness hands the lock to a waiter only about once per 0.5 ms, while the parse thread relocks back-to-back between chunks. Completing one pull needs every one of these 5-6 acquisitions to independently win that fairness lottery before the parse thread reclaims the lock, which matches the observed 4-34 fps spread under a sustained flood. Candidate fix: acquire the lock once per drain cycle and hold it across drain, fan-out and ship, instead of the current lock/unlock sequence. That cuts the number of required fairness wins from 5-6 to 1 per pull.

natsukium referenced this issue from a commit

2026-09-24 13:19:28 +09:00

daemon: take the parse lock once per grid cycle

natsukium referenced this issue from a commit

2026-09-24 15:22:58 +09:00

daemon: pin that the run loop yields between flood drains

natsukium referenced this issue from a pull request that will close it,

2026-09-24 16:23:25 +09:00

daemon: paint at the display rate under a sustained flood #380

natsukium closed this issue

2026-09-24 18:37:25 +09:00

natsukium referenced this issue from a commit