grid, daemon: ship scrolls over written rows and DL/IL as directives #455

Merged

natsukium merged 4 commits from scroll-447 into main 2026-09-25 16:10:36 +09:00

Conversation 0 Commits 4 Files changed 23 +1041 -227

natsukium commented 2026-09-25 14:55:13 +09:00

Owner

Copy link

The daemon now sends GridMsg::Scrolled for a scroll over rows already written in the same frame, and for IL / DL. The directive is followed by only the rows the shift does not reproduce, so the daemon no longer resends the whole scrolled region as rows. The wire format is unchanged, and so is felis.proto.

What changed

• grid: rows written before a scroll move with it. scroll_region_up/down always queue the directive, and Damage::shift_band moves the band's dirty bits along with the rows and marks the rows the scroll vacated. A row written before the scroll therefore stays owed at its new position. IL / DL rotate the band from the cursor row to the bottom margin and send the same directive. They used to call mark_all(). The grid's effect queue folds consecutive shifts of one band into one entry. That entry carries the range of the scroll sequence it covers, first_seq..=last_seq. A scrollback push also folds past a trailing directive, so a line-feed flood drains as two effects.
• daemon: each subscriber moves its owed rows too. In the old code, accept_scroll fell back to replaying the band when a subscriber still owed rows in it. Now it shifts the subscriber's own Damage the same way. It folds consecutive shifts of one band, and it drops a directive once the whole band is owed anyway. The 1024 queue cap stays.
  • Soundness guards: offer_scroll replays every row when a folded entry spans a full replay, so the shift is never applied twice. catch_up_to replays every row when the grid's damage would merge in while a scroll is still queued in the grid. grid_cycle_for, the path that answers a pull, now drains effects first, which keeps that second case rare during a flood.
• Compatibility with released clients (see below): each vacated row that is resent with non-blank content is preceded by an entry of default cells.
• grid: a row write can land on storage past the row's watermark. ScreenBuffer::write_row_cells compared the incoming cells with the row's storage past its watermark and skipped the write when they matched. That storage is where a directive leaves the rotated-out cells. So on the client, a line that matched the line that had just scrolled out showed blank. main has this bug for pure scrolls today: a loop printing \ny shows it once the screen is full.
• grid: DECSTBM no longer marks every row. DECSTBM changes no cell, and the damage contract already keeps cursor-only moves out of the tracker. vim scrolls with ESC[1;Nr ESC[H ESC[M ESC[r, and the two DECSTBMs in it turned every ^E into a full-screen replay.
• Tests:
  • damage_correctness.rs used to count a scrolled band as covered. It now applies the queued directives to the prior screen and checks every row that is not marked, so the test no longer passes trivially whenever a band scrolled.
  • The grid-level mirror property in proptest_grid.rs now starts from a non-empty screen.
  • proptest_damage.rs models shift_band / all_dirty_in_range.
  • The row-cache property can now apply up to three ops per frame.
  • New daemon property: every_mirror_ends_each_compose_holding_the_grids_cells. It drives two subscribers into real ShadowScreens, each pulling at its own random cadence, through random output (DECSTBM, IL / DL, DECSLRM, the alt screen, scrollback clears, BCE), drain cadences, resizes and scrollback browsing. After every compose it requires each shadow to hold exactly the grid's cells. I ran it at 512 cases in the suite and at 30 000 in release.
• Docs:
  • docs/reference/ipc.md (the Scrolled / RowDelta bullets and the send-side rules).
  • docs/explanation/rendering/damage-tracking.md: the mechanism, plus records for the decisions below.
  • The docs/explanation/rendering/pipeline.md coalescing sentence, which described a merge the code did not do.
  • docs/reference/testing.md.
  • CHANGELOG.md.

Open questions from the issue

• Coalescing across bands: each band keeps its own directive. There is no fallback to a replay after some count, because a directive costs a few bytes and a row about two bytes per column. The existing 1024-per-subscriber cap still sends the band as rows. Recorded in damage-tracking.md with a "Revisit if".
• Where the rotation lives: Damage learned shift_band (plus all_dirty_in_range / is_dirty). The grid calls it right after rotate_region, and the subscriber stream calls it in accept_scroll. damage_correctness.rs (now strict) and the row-cache property both pass with a directive and rows in the same cycle.
• DECSLRM sub-rectangles: they stay on replay. Scrolled has no column bounds, and no measured workload scrolls a DECSLRM band every frame. Recorded with a "Revisit if".
• Scrollback-anchored placements: unchanged. DL / IL still use rotate_region without a scrollback push, and PtyEffect::Scrolled is only parked in the queue. A scrollback push folds past a trailing directive, which commutes with it because the directive touches no placement.
• Remeasurement: see below.

Released client (v0.1.0)

v0.1.0 is not a tag on the remote. I compared against v0.1.0-rc.6 (4b9af186), the code released as 0.1.0. Between rc.6 and main, the client's apply path (ShadowScreen::apply / scroll, ScreenBuffer::apply_scroll_directive and write_row_cells) differs only by an unrelated clear_damage.

• Checked by reasoning: a directive is admitted by the same bounds, it applies the rotation with default cells, and RowDelta entries override the rows in order. So a stream in which "every row not resent equals the shifted mirror" converges, and nothing on the client assumes the band is clean afterwards.
• Checked by test: the one rc.6 behaviour that could break convergence is the write_row_cells skip described above. Sending directives far more often would have hit it on every yes line. The daemon property test checks every row entry against that rc.6 behaviour: it fails if a row's storage equals the incoming cells while the row reads differently. The test fails when the default-cell entry is removed and passes with it.

Measurements

Setup: Linux, NVIDIA, release build, 212x93 window. The client was instrumented to count, per pull cycle, the rows and bytes it receives in RowDelta and the Scrolled directives. Base is origin/main at 35cd16b8. There were 3 runs per workload per build, and the table gives medians of per-cycle values. One-minute load average during the runs:

• vim and flood: 1.3–4.2.
• pager: 4.2–4.9.
• less and key: 4.7–11.1, because a CI job started.

The row and directive counts do not depend on load.

workload	rows / cycle (base → new)	bytes / cycle (base → new)	Scrolled per cycle (base → new)
less, j every 30 ms	93 → 3	21.6 KB → 0.69 KB	0 → 1.00
a line every 33 ms (printf "\n...")	93 → 3	38.9 KB → 1.07 KB	0 → 1.00
vim -u NONE, ^E every 30 ms	93 → 5 (mean 48 → 4)	22.5 KB → 1.6 KB	0 → 0.50
seq -w flood	93 → 93	22.2 KB → 22.2 KB	0 → 0
one keystroke every 50 ms	1 → 1	0.30 KB → 0.30 KB	0 → 0

• Flood: it scrolls thousands of lines per frame, so the band is shifted by its full height and every row is new. The directive is dropped by design.
• Why 3 rows in less and the pager: the new bottom row, the default-cell entry before it, and the row the \r before each line feed marked. The last one is a pre-existing overdraw from cursor-only moves, filed as #454.

Parse-thread cost, felis-grid Criterion benches on the same host with a load average of about 3–5. Base → new:

• scroll_region/full_screen_scroll_up_1: 55.0 → 19.9 ns
• partial_region_scroll_up_1: 111 → 65.6 ns
• mixed_print_scroll_full: 131.7 → 118.2 ns
• flood_scroll_up_1: 42.1 → 34.5 ns
• ascii_throughput/crlf_lines: 101 → 98 µs
• bare_lf_staircase: 134 → 113 µs

The full-screen and partial-region cases gain because a directive now folds into the tail entry of the queue instead of adding one entry per line.

Review

Independent review with pi (sol) over origin/main...HEAD:

• Round 1: PASS, with two non-blocking findings.
  • ipc.md said vacated rows that are default blanks are left out. In fact every vacated row is sent again. Fixed.
  • The post-change measurements were missing. Added: the table above, plus CHANGELOG.md.
• Round 2: PASS, with no findings. This round covered the DECSTBM commit and the fast path in shift_band.

Closes #447

The daemon now sends `GridMsg::Scrolled` for a scroll over rows already written in the same frame, and for `IL` / `DL`. The directive is followed by only the rows the shift does not reproduce, so the daemon no longer resends the whole scrolled region as rows. The wire format is unchanged, and so is `felis.proto`. ## What changed - **grid: rows written before a scroll move with it.** `scroll_region_up`/`down` always queue the directive, and `Damage::shift_band` moves the band's dirty bits along with the rows and marks the rows the scroll vacated. A row written before the scroll therefore stays owed at its new position. `IL` / `DL` rotate the band from the cursor row to the bottom margin and send the same directive. They used to call `mark_all()`. The grid's effect queue folds consecutive shifts of one band into one entry. That entry carries the range of the scroll sequence it covers, `first_seq..=last_seq`. A scrollback push also folds past a trailing directive, so a line-feed flood drains as two effects. - **daemon: each subscriber moves its owed rows too.** In the old code, `accept_scroll` fell back to replaying the band when a subscriber still owed rows in it. Now it shifts the subscriber's own `Damage` the same way. It folds consecutive shifts of one band, and it drops a directive once the whole band is owed anyway. The 1024 queue cap stays. - Soundness guards: `offer_scroll` replays every row when a folded entry spans a full replay, so the shift is never applied twice. `catch_up_to` replays every row when the grid's damage would merge in while a scroll is still queued in the grid. `grid_cycle_for`, the path that answers a pull, now drains effects first, which keeps that second case rare during a flood. - **Compatibility with released clients (see below):** each vacated row that is resent with non-blank content is preceded by an entry of default cells. - **grid: a row write can land on storage past the row's watermark.** `ScreenBuffer::write_row_cells` compared the incoming cells with the row's storage past its watermark and skipped the write when they matched. That storage is where a directive leaves the rotated-out cells. So on the client, a line that matched the line that had just scrolled out showed blank. `main` has this bug for pure scrolls today: a loop printing `\ny` shows it once the screen is full. - **grid: DECSTBM no longer marks every row.** DECSTBM changes no cell, and the damage contract already keeps cursor-only moves out of the tracker. vim scrolls with `ESC[1;Nr ESC[H ESC[M ESC[r`, and the two DECSTBMs in it turned every `^E` into a full-screen replay. - **Tests:** - `damage_correctness.rs` used to count a scrolled band as covered. It now applies the queued directives to the prior screen and checks every row that is not marked, so the test no longer passes trivially whenever a band scrolled. - The grid-level mirror property in `proptest_grid.rs` now starts from a non-empty screen. - `proptest_damage.rs` models `shift_band` / `all_dirty_in_range`. - The row-cache property can now apply up to three ops per frame. - New daemon property: `every_mirror_ends_each_compose_holding_the_grids_cells`. It drives two subscribers into real `ShadowScreen`s, each pulling at its own random cadence, through random output (DECSTBM, `IL` / `DL`, DECSLRM, the alt screen, scrollback clears, BCE), drain cadences, resizes and scrollback browsing. After every compose it requires each shadow to hold exactly the grid's cells. I ran it at 512 cases in the suite and at 30 000 in release. - **Docs:** - `docs/reference/ipc.md` (the `Scrolled` / `RowDelta` bullets and the send-side rules). - `docs/explanation/rendering/damage-tracking.md`: the mechanism, plus records for the decisions below. - The `docs/explanation/rendering/pipeline.md` coalescing sentence, which described a merge the code did not do. - `docs/reference/testing.md`. - `CHANGELOG.md`. ## Open questions from the issue - **Coalescing across bands:** each band keeps its own directive. There is no fallback to a replay after some count, because a directive costs a few bytes and a row about two bytes per column. The existing 1024-per-subscriber cap still sends the band as rows. Recorded in damage-tracking.md with a "Revisit if". - **Where the rotation lives:** `Damage` learned `shift_band` (plus `all_dirty_in_range` / `is_dirty`). The grid calls it right after `rotate_region`, and the subscriber stream calls it in `accept_scroll`. `damage_correctness.rs` (now strict) and the row-cache property both pass with a directive and rows in the same cycle. - **DECSLRM sub-rectangles:** they stay on replay. `Scrolled` has no column bounds, and no measured workload scrolls a DECSLRM band every frame. Recorded with a "Revisit if". - **Scrollback-anchored placements:** unchanged. `DL` / `IL` still use `rotate_region` without a scrollback push, and `PtyEffect::Scrolled` is only parked in the queue. A scrollback push folds past a trailing directive, which commutes with it because the directive touches no placement. - **Remeasurement:** see below. ## Released client (v0.1.0) `v0.1.0` is not a tag on the remote. I compared against `v0.1.0-rc.6` (4b9af186), the code released as 0.1.0. Between rc.6 and `main`, the client's apply path (`ShadowScreen::apply` / `scroll`, `ScreenBuffer::apply_scroll_directive` and `write_row_cells`) differs only by an unrelated `clear_damage`. - **Checked by reasoning:** a directive is admitted by the same bounds, it applies the rotation with default cells, and `RowDelta` entries override the rows in order. So a stream in which "every row not resent equals the shifted mirror" converges, and nothing on the client assumes the band is clean afterwards. - **Checked by test:** the one rc.6 behaviour that could break convergence is the `write_row_cells` skip described above. Sending directives far more often would have hit it on every `yes` line. The daemon property test checks every row entry against that rc.6 behaviour: it fails if a row's storage equals the incoming cells while the row reads differently. The test fails when the default-cell entry is removed and passes with it. ## Measurements Setup: Linux, NVIDIA, release build, 212x93 window. The client was instrumented to count, per pull cycle, the rows and bytes it receives in `RowDelta` and the `Scrolled` directives. Base is `origin/main` at 35cd16b8. There were 3 runs per workload per build, and the table gives medians of per-cycle values. One-minute load average during the runs: - vim and flood: 1.3–4.2. - pager: 4.2–4.9. - less and key: 4.7–11.1, because a CI job started. The row and directive counts do not depend on load. | workload | rows / cycle (base → new) | bytes / cycle (base → new) | `Scrolled` per cycle (base → new) | | --- | --- | --- | --- | | `less`, `j` every 30 ms | 93 → 3 | 21.6 KB → 0.69 KB | 0 → 1.00 | | a line every 33 ms (`printf "\n..."`) | 93 → 3 | 38.9 KB → 1.07 KB | 0 → 1.00 | | `vim -u NONE`, `^E` every 30 ms | 93 → 5 (mean 48 → 4) | 22.5 KB → 1.6 KB | 0 → 0.50 | | `seq -w` flood | 93 → 93 | 22.2 KB → 22.2 KB | 0 → 0 | | one keystroke every 50 ms | 1 → 1 | 0.30 KB → 0.30 KB | 0 → 0 | - **Flood:** it scrolls thousands of lines per frame, so the band is shifted by its full height and every row is new. The directive is dropped by design. - **Why 3 rows in less and the pager:** the new bottom row, the default-cell entry before it, and the row the `\r` before each line feed marked. The last one is a pre-existing overdraw from cursor-only moves, filed as #454. Parse-thread cost, felis-grid Criterion benches on the same host with a load average of about 3–5. Base → new: - `scroll_region/full_screen_scroll_up_1`: 55.0 → 19.9 ns - `partial_region_scroll_up_1`: 111 → 65.6 ns - `mixed_print_scroll_full`: 131.7 → 118.2 ns - `flood_scroll_up_1`: 42.1 → 34.5 ns - `ascii_throughput/crlf_lines`: 101 → 98 µs - `bare_lf_staircase`: 134 → 113 µs The full-screen and partial-region cases gain because a directive now folds into the tail entry of the queue instead of adding one entry per line. ## Review Independent review with `pi` (sol) over `origin/main...HEAD`: - **Round 1: PASS**, with two non-blocking findings. - `ipc.md` said vacated rows that are default blanks are left out. In fact every vacated row is sent again. Fixed. - The post-change measurements were missing. Added: the table above, plus `CHANGELOG.md`. - **Round 2: PASS**, with no findings. This round covered the DECSTBM commit and the fast path in `shift_band`. Closes #447

natsukium added 4 commits 2026-09-25 14:55:13 +09:00

grid: land a row write over storage past the row's watermark ... 5c97a7c8ea

A scroll directive recycles the rotated-out rows as the vacated band
and only drops their watermark, leaving the old cells as unread
storage. `write_row_cells` compared the incoming row with that storage
and skipped the write on a match, so a vacated row restated with the
content that had just scrolled out read blank on the client. A shell
loop printing `\ny` shows it on every line once the screen is full.
The early return now requires the whole row to be below the watermark.

Assisted-by: Claude Code

render: let the row-cache property apply several ops per frame ... c4956cf9b8

With one op per frame a scroll directive never lands on a band a
write already marked in the same frame, which is the common case once
the daemon ships a directive for a scroll over written rows. Up to
three ops now share a frame.

Assisted-by: Claude Code

grid, daemon: ship scrolls over written rows and DL/IL as directives ... 6453c3d00e

The daemon sent `GridMsg::Scrolled` only when no row of the scrolled
band was dirty, and never for DL / IL, which marked the whole grid.
Producers almost never scroll a clean band: a flood or a streaming
printer writes its line before the line feed, a pager rewrites its
prompt first, and vim scrolls with DECSTBM plus DL. On a 212x93 window
the flood, pager, less and vim ^E workloads received almost no
directives and replayed 92 or 93 rows per frame.

A scroll now shifts the band's dirty bits with the rows and marks the
vacated rows, so a row written before it stays owed at its new place
and the directive is sound whatever the band held. DL / IL rotate the
band from the cursor row to the bottom margin and ship the same
directive. The grid folds consecutive shifts of one band into one
effect (the entry carries the span of scroll sequence it covers, so a
subscriber whose full replay landed inside that span replays again
instead of shifting twice), and a line-feed flood drains as one
scrollback push plus one directive.

Each subscriber shifts its own owed rows the same way when it takes a
directive, instead of downgrading to a band replay when it is behind;
it folds shifts of one band, drops a directive whose band ends up owed
whole, and still caps the queue at 1024. A fan-out that would merge
grid damage while a scroll is still queued in the grid replays every
row instead, since that damage already sits in post-shift coordinates.
The pull-answering cycle now drains effects first so that case stays
rare under a flood.

A 0.1.0 client compares a row write with the storage past the row's
watermark, which a directive leaves holding the rotated-out cells, and
skips the write on a match. So a vacated row restated with anything
but blanks is preceded by an entry of default cells for that row:
whichever of the two differs from the storage lands. The wire format
is unchanged.

The no-underdraw harness now shifts the prior screen by the queued
directives instead of treating a scrolled band as covered, which had
made it vacuous for any band that scrolled. A new daemon property test
drives two subscribers at random pull cadences through resizes,
scrollback browsing, DECSTBM, DECSLRM and the alternate screen into
real shadows, and checks every row entry against the 0.1.0 row write.

Assisted-by: Claude Code

grid: leave the damage alone when the scroll margins change ... a53fc7c0f7

DECSTBM moves the margins and homes the cursor; it changes no cell,
and the damage contract already keeps cursor-only moves out of the
tracker. Marking every row there turned vim's scroll (DECSTBM, DL,
then DECSTBM again to reset) into a replay of the whole screen, so
the DL directive never reached the client.

Assisted-by: Claude Code

natsukium force-pushed scroll-447 from a53fc7c0f7 to 76ec65ab54 2026-09-25 15:14:54 +09:00 Compare

natsukium merged commit 5f5e495778 into main 2026-09-25 16:10:36 +09:00

natsukium deleted branch scroll-447 2026-09-25 16:10:36 +09:00

natsukium referenced this pull request from a commit 2026-09-25 16:10:38 +09:00

grid, daemon: ship scrolls over written rows and DL/IL as directives (#455)

natsukium referenced this pull request 2026-09-25 19:00:17 +09:00

bench: follow up the 9d1d842f cross-terminal run #405

natsukium referenced this pull request 2026-09-25 19:42:17 +09:00

grid: stop marking the cursor row on cursor-only moves #463

natsukium referenced this pull request from a commit 2026-09-25 20:35:43 +09:00

grid: stop marking the cursor row on cursor-only moves (#463)

natsukium referenced this pull request 2026-09-26 10:01:52 +09:00

changelog: fold the pre-release entries into the 0.1.0 section #476

natsukium referenced this pull request from a commit 2026-09-26 10:59:25 +09:00

changelog: fold the pre-release entries into the 0.1.0 section (#476)

Commenting is not possible because the repository is archived.

Reviewers

No reviewers

Labels

Clear labels   

priority/P0

Must be resolved before the v0.1.0 tag

  

priority/P1

Must be settled before the first public contract freezes

  

priority/P2

Explicitly deferred until after v0.1.0

  

release/v0.1.0

Work tracked for the first tagged release

  

status/blocked

Cannot start now: waits on an upstream release, another issue, or a recorded revisit trigger

  

status/planned

Triaged with a source-grounded plan and a passed pi review; ready to implement

  

type/bug

Shipped behavior that contradicts the docs or the code's own intent; needs a product code change

  

type/design

An open scope or trade-off question; needs a recorded decision in the owning explanation doc before code

  

type/test-gap

Documented or reviewed behavior that no test pins; usually split from a review finding, needs no product code change

  

type/tracker

Parent issue that tracks child work

No labels

priority/P0

priority/P1

priority/P2

release/v0.1.0

status/blocked

status/planned

type/bug

type/design

type/test-gap

type/tracker

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

natsukium/felis!455

No description provided.