cli: mouse input for driving a fullscreen TUI from felis sessions #131

New issue

Closed

opened 2026-10-12 23:05:59 +09:00 by natsukium · 0 comments

natsukium commented

2026-10-12 23:05:59 +09:00

Owner

Problem

An agent driving a fullscreen TUI through felis sessions can read a button's label off the screen (search --source visible returns its col_spans), but cannot click it. Pressing the label it sees is often one step where finding the equivalent keybinding is a help-screen hunt or impossible, and mouse-first TUIs are increasingly common. The same gap blocks an agent verifying the mouse handling of a TUI it is building.

Today the caller hand-encodes SGR 1006 reports and sends them with --raw:

printf '\033[<0;13;12M\033[<0;13;12m' | felis sessions send "$ID" --raw -   # press, release

This guesses the encoding: it is wrong under X10 or SGR-pixel (?1016) reporting, and if the application enabled no mouse tracking at all the bytes reach it as garbage input, with nothing telling the caller. SGR-pixel is not a corner case: Textual enables ?1016 on every terminal that supports ?2048, felis included. This is the stale-mode problem send --key solves for keys by encoding daemon-side (docs/explanation/architecture/control-surfaces.md "send --key reuses the chord grammar and sends the keystroke itself").

No task was found that is reachable only by mouse (Claude Code's one mouse-only gap, starting a selection, does not matter to an agent that reads the screen); the consumer is the click-what-you-read path above and TUI mouse verification.

What exists

Input::Mouse(MouseEvent { button, action, mods, x, y, px, py }) (1-based cells and pixels) is accepted from any attachment, window or not, since minor 0. The daemon encodes it against the session's current mouse protocol and encoding (felis_grid::encode_mouse). It reserves MAX_MOUSE_REPORT_BYTES and waits for budget rather than being dropped (serve.rs input_reservation_bytes).
send already attaches, drains the Ops rehydrate burst, writes its frames in order, and confirms admission with Session::InputFence. The burst carries GridMsg::ModeFlags.mouse_protocol (not the encoding) and no grid rows; the grid dims arrive in the attach reply (Attached.info.dims), which attach_by_prefix currently discards.
The fence confirms that the daemon admitted the frames, not that they encoded to bytes.
The daemon holds the window-supplied pixel extents (TIOCGWINSZ, cell_pixel_w); a session no window has sized holds 0 until #143 gives it a nominal cell size.

Design

Surface: send --mouse EVENT, repeatable. No new verb. send's explanation keeps --key on send because one call names one target, resolves it once, and states the input sequence in order; that holds for mouse events unchanged. --mouse depends on no other flag, so it respects the frozen-grammar rule ("no new flag may depend on another"); the first constraint becomes "at least one of TEXT, --key, --mouse or --wait". The control-surfaces sentence "a different payload kind is a new verb" contradicts --key already being a payload kind on send, and is reworded to what it guards: a flag that reads only in combination with another.

Grammar. EVENT := [MODS+]ACTION[-BUTTON]:COL,ROW

ACTION: click (press then release), press, drag, release, wheel-up, wheel-down, wheel-left, wheel-right. click exists because a press whose release the caller forgets leaves the TUI mid-drag.
BUTTON: left (default), middle, right; not accepted on wheel actions.
MODS: shift, alt, ctrl in the [keymap] chord spelling; super is a usage error (no mouse report carries it).
COL,ROW: 0-based, matching search's col_spans and capture's live row, so a match is clicked without arithmetic. The CLI adds 1.
Bare motion (?1003 hover) is left out until a consumer needs it.

Ordering. TEXT is pasted first (unchanged); --key and --mouse frames then go out interleaved in argv order. Today's "keys after text" stays true.

Pixels: derived daemon-side. A CLI event carries px = py = 0, which no 1-based pixel coordinate uses. When the encoding is SgrPixels and an event carries no pixels, the encoder fills them with the center of the event's cell, computed from the full pixel extents the session reports to the program (TIOCGWINSZ, and ?2048 once #143 settles it) and the grid dims: px = floor((col0 + 0.5) * pixel_w / cols) + 1, likewise for py. Not from the truncated cell_pixel_w: a program maps pixels back with the full ratio (Textual: (px - 1) / (pixel_w / cols)), and on an 809-pixel, 80-column grid a center built from cell_pixel_w = 10 lands column 80's click on column 79. Tests cover extents not divisible by the cell counts. Depends on #143, for two of its outcomes: ?2048 reports the PTY's real pixel extents (until then a Textual app crashes on any mouse event, CLI or window), and a session no window has sized gets a nominal cell size, so the headless sessions spawn session this verb mostly drives always has pixel geometry to derive from.

Preflight: refuse when the program is not listening. After the drain, the CLI checks every event against ModeFlags.mouse_protocol with the filter encode_mouse applies (factored into a shared pure predicate): protocol Off, or a drag under ?1000. Any filtered event, or a coordinate outside Attached.info.dims (kept instead of discarded), fails the whole command with nothing written (exit 1; error kind to settle: invalid_request vs a new kind).

What a success guarantees, and what is accepted. Success means the program had mouse reporting on for these events when the CLI looked, and the daemon admitted all of them. Not guaranteed, and accepted rather than designed out:

a mode change between the drain and the write (including one the command's own TEXT, keys, or press provokes): an event the new mode filters is dropped while the fence still succeeds. Programs set their mouse modes once at start-up;
X10 (no ?1006) clamps a coordinate above 223 onto the edge cell; only a legacy program on a grid over 223 cells reaches it.

Closing these would need a daemon-side per-event outcome on the fence, a wire change no consumer has needed.

Admission. No new mechanism. Each event reserves like today and the trailing InputFence confirms the daemon admitted all of them. A press can still land without its release: when the fence times out because the child stopped reading stdin, when the session ends between the two, or when the press itself makes the program turn mouse reporting off (the accepted mode-change case above).

Version skew. No wire change and no minor gate: Input::Mouse from a non-window attachment and ModeFlags.mouse_protocol both exist since minor 0.

Not in scope. No bridge op: sessions.send does not take --key either, and no bridge client has asked.

Cascade

docs/reference/cli.md "Verb details" send; control-surfaces.md (send sections, the reworded sentence, and the control-surface name map's send --key entry gains send --mouse).
docs/reference/ipc.md "Input (kind = 1)": the Mouse(MouseEvent) description (pixels are client-supplied only for a window; a non-window sender's 0 pixels are derived daemon-side), and InputFence: the fence confirms admission, not delivery to the PTY.
docs/reference/protocols/vt-compliance.md "Mouse and focus" and docs/reference/protocols/key-encoding.md: ?1016 reports from a non-window sender carry cell-center pixels, not the pointer's true position.
skills/felis/SKILL.md: replace the "aim a mouse report (1-based)" paragraph with a --mouse example fed from search --source visible.
CHANGELOG.md entry.

## Problem An agent driving a fullscreen TUI through `felis sessions` can read a button's label off the screen (`search --source visible` returns its `col_spans`), but cannot click it. Pressing the label it sees is often one step where finding the equivalent keybinding is a help-screen hunt or impossible, and mouse-first TUIs are increasingly common. The same gap blocks an agent verifying the mouse handling of a TUI it is building. Today the caller hand-encodes SGR 1006 reports and sends them with `--raw`: ```sh printf '\033[<0;13;12M\033[<0;13;12m' | felis sessions send "$ID" --raw - # press, release ``` This guesses the encoding: it is wrong under X10 or SGR-pixel (`?1016`) reporting, and if the application enabled no mouse tracking at all the bytes reach it as garbage input, with nothing telling the caller. SGR-pixel is not a corner case: Textual enables `?1016` on every terminal that supports `?2048`, felis included. This is the stale-mode problem `send --key` solves for keys by encoding daemon-side (`docs/explanation/architecture/control-surfaces.md` "`send --key` reuses the chord grammar and sends the keystroke itself"). No task was found that is reachable *only* by mouse (Claude Code's one mouse-only gap, starting a selection, does not matter to an agent that reads the screen); the consumer is the click-what-you-read path above and TUI mouse verification. ## What exists - `Input::Mouse(MouseEvent { button, action, mods, x, y, px, py })` (1-based cells and pixels) is accepted from any attachment, window or not, since minor 0. The daemon encodes it against the session's current mouse protocol and encoding (`felis_grid::encode_mouse`). It reserves `MAX_MOUSE_REPORT_BYTES` and waits for budget rather than being dropped (`serve.rs` `input_reservation_bytes`). - `send` already attaches, drains the Ops rehydrate burst, writes its frames in order, and confirms admission with `Session::InputFence`. The burst carries `GridMsg::ModeFlags.mouse_protocol` (not the encoding) and no grid rows; the grid dims arrive in the attach reply (`Attached.info.dims`), which `attach_by_prefix` currently discards. - The fence confirms that the daemon admitted the frames, not that they encoded to bytes. - The daemon holds the window-supplied pixel extents (`TIOCGWINSZ`, `cell_pixel_w`); a session no window has sized holds `0` until #143 gives it a nominal cell size. ## Design **Surface: `send --mouse EVENT`, repeatable. No new verb.** `send`'s explanation keeps `--key` on `send` because one call names one target, resolves it once, and states the input sequence in order; that holds for mouse events unchanged. `--mouse` depends on no other flag, so it respects the frozen-grammar rule ("no new flag may depend on another"); the first constraint becomes "at least one of TEXT, `--key`, `--mouse` or `--wait`". The control-surfaces sentence "a different payload kind is a new verb" contradicts `--key` already being a payload kind on `send`, and is reworded to what it guards: a flag that reads only in combination with another. **Grammar.** `EVENT := [MODS+]ACTION[-BUTTON]:COL,ROW` - `ACTION`: `click` (press then release), `press`, `drag`, `release`, `wheel-up`, `wheel-down`, `wheel-left`, `wheel-right`. `click` exists because a press whose release the caller forgets leaves the TUI mid-drag. - `BUTTON`: `left` (default), `middle`, `right`; not accepted on wheel actions. - `MODS`: `shift`, `alt`, `ctrl` in the `[keymap]` chord spelling; `super` is a usage error (no mouse report carries it). - `COL,ROW`: **0-based**, matching `search`'s `col_spans` and `capture`'s live `row`, so a match is clicked without arithmetic. The CLI adds 1. - Bare motion (`?1003` hover) is left out until a consumer needs it. **Ordering.** TEXT is pasted first (unchanged); `--key` and `--mouse` frames then go out interleaved in argv order. Today's "keys after text" stays true. **Pixels: derived daemon-side.** A CLI event carries `px = py = 0`, which no 1-based pixel coordinate uses. When the encoding is `SgrPixels` and an event carries no pixels, the encoder fills them with the center of the event's cell, computed from the full pixel extents the session reports to the program (`TIOCGWINSZ`, and `?2048` once #143 settles it) and the grid dims: `px = floor((col0 + 0.5) * pixel_w / cols) + 1`, likewise for `py`. Not from the truncated `cell_pixel_w`: a program maps pixels back with the full ratio (Textual: `(px - 1) / (pixel_w / cols)`), and on an 809-pixel, 80-column grid a center built from `cell_pixel_w = 10` lands column 80's click on column 79. Tests cover extents not divisible by the cell counts. **Depends on #143**, for two of its outcomes: `?2048` reports the PTY's real pixel extents (until then a Textual app crashes on any mouse event, CLI or window), and a session no window has sized gets a nominal cell size, so the headless `sessions spawn` session this verb mostly drives always has pixel geometry to derive from. **Preflight: refuse when the program is not listening.** After the drain, the CLI checks every event against `ModeFlags.mouse_protocol` with the filter `encode_mouse` applies (factored into a shared pure predicate): protocol `Off`, or a `drag` under `?1000`. Any filtered event, or a coordinate outside `Attached.info.dims` (kept instead of discarded), fails the whole command with nothing written (exit `1`; error kind to settle: `invalid_request` vs a new kind). **What a success guarantees, and what is accepted.** Success means the program had mouse reporting on for these events when the CLI looked, and the daemon admitted all of them. Not guaranteed, and accepted rather than designed out: - a mode change between the drain and the write (including one the command's own TEXT, keys, or press provokes): an event the new mode filters is dropped while the fence still succeeds. Programs set their mouse modes once at start-up; - X10 (no `?1006`) clamps a coordinate above 223 onto the edge cell; only a legacy program on a grid over 223 cells reaches it. Closing these would need a daemon-side per-event outcome on the fence, a wire change no consumer has needed. **Admission.** No new mechanism. Each event reserves like today and the trailing `InputFence` confirms the daemon admitted all of them. A press can still land without its release: when the fence times out because the child stopped reading stdin, when the session ends between the two, or when the press itself makes the program turn mouse reporting off (the accepted mode-change case above). **Version skew.** No wire change and no minor gate: `Input::Mouse` from a non-window attachment and `ModeFlags.mouse_protocol` both exist since minor 0. **Not in scope.** No bridge op: `sessions.send` does not take `--key` either, and no bridge client has asked. ## Cascade - `docs/reference/cli.md` "Verb details" `send`; `control-surfaces.md` (`send` sections, the reworded sentence, and the control-surface name map's `send --key` entry gains `send --mouse`). - `docs/reference/ipc.md` "Input (kind = 1)": the `Mouse(MouseEvent)` description (pixels are client-supplied only for a window; a non-window sender's `0` pixels are derived daemon-side), and `InputFence`: the fence confirms admission, not delivery to the PTY. - `docs/reference/protocols/vt-compliance.md` "Mouse and focus" and `docs/reference/protocols/key-encoding.md`: `?1016` reports from a non-window sender carry cell-center pixels, not the pointer's true position. - `skills/felis/SKILL.md`: replace the "aim a mouse report (1-based)" paragraph with a `--mouse` example fed from `search --source visible`. - `CHANGELOG.md` entry.

natsukium added the

Area/cli

Area/daemon

Kind/Feature

labels

2026-10-13 09:55:42 +09:00

natsukium referenced this issue

2026-10-13 23:10:46 +09:00

vt: Textual apps crash on the first mouse event because ?2048 reports 0 pixel extents #143

natsukium added the

Area/grid

label