Add an offscreen Renderer with RGBA readback and pinned fonts for headless consumers #441

Closed

opened 2026-09-25 12:07:55 +09:00 by natsukium · 1 comment

natsukium commented

2026-09-25 12:07:55 +09:00

Owner

Motivation

felis-reel is a planned downstream tool that renders a felis-fcast .fcast recording to MP4, WebM or GIF headlessly, playing the role agg plays for asciinema. It should draw through felis-render-wgpu so the output looks like a real felis window. Today it cannot: Renderer can only be built on a window surface.

Both constructors require T: HasWindowHandle + HasDisplayHandle, call create_surface, pick the adapter with compatible_surface: Some(&surface), and take the texture format from the surface (crates/felis-render-wgpu/src/lib.rs:370-449).
render() ends in get_current_texture() / present() (lib.rs:999-1023).
Font selection always goes through FontStack::auto_discover unless a Warmup supplies the stack (lib.rs:539-547). The same input therefore renders differently on two machines.

docs/explanation/rendering/pipeline.md:262 already anticipates a "recorder, relay, headless test client" consumer.

Proposal

Offscreen target. Make the render target an enum, Surface(Surface) or Offscreen { texture, format }, and add Renderer::new_offscreen(width, height, cfg, warmup). It selects the adapter with compatible_surface: None, as probe.rs:23 and Warmup::start_gpu() already do, and uses a fixed Rgba8UnormSrgb format. encode_cell_pass and encode_post_pass already take any &TextureView.
Pixel readback. Add Renderer::read_frame(&mut self, out: &mut Vec<u8>). It returns tightly packed RGBA of the post-processed frame through a reusable, row-padded staging buffer. capture_last_frame() (capture.rs:38) already reads back an offscreen texture, but it returns only a histogram, skips the post pass, and allocates on every call. It could become a thin wrapper over read_frame.
Pinned fonts. Let Warmup or RendererConfig accept a preloaded FontStack (FontStack::new, felis-shaping/src/lib.rs:354), so the caller controls exactly which font files are used.

Scope

Only "draw into memory and hand back pixels" belongs in felis. Video encoding, frame timing and the .fcast format stay in felis-reel. The smoke test and any screenshot-based golden tests could also use the new API.

Rejected alternatives (from the felis-reel side)

A separate CPU rasterizer on felis-shaping. It would duplicate glyph placement, decorations, cursor shapes, Kitty image placement, reverse video and the post shader, and its output would drift from the real window.
Screen-capturing felis-client attached to felis-fcast serve. This needs a display, runs only in real time, and does not produce deterministic output.

## Motivation felis-reel is a planned downstream tool that renders a felis-fcast `.fcast` recording to MP4, WebM or GIF headlessly, playing the role `agg` plays for asciinema. It should draw through felis-render-wgpu so the output looks like a real felis window. Today it cannot: `Renderer` can only be built on a window surface. - Both constructors require `T: HasWindowHandle + HasDisplayHandle`, call `create_surface`, pick the adapter with `compatible_surface: Some(&surface)`, and take the texture format from the surface (`crates/felis-render-wgpu/src/lib.rs:370-449`). - `render()` ends in `get_current_texture()` / `present()` (`lib.rs:999-1023`). - Font selection always goes through `FontStack::auto_discover` unless a `Warmup` supplies the stack (`lib.rs:539-547`). The same input therefore renders differently on two machines. `docs/explanation/rendering/pipeline.md:262` already anticipates a "recorder, relay, headless test client" consumer. ## Proposal 1. **Offscreen target.** Make the render target an enum, `Surface(Surface)` or `Offscreen { texture, format }`, and add `Renderer::new_offscreen(width, height, cfg, warmup)`. It selects the adapter with `compatible_surface: None`, as `probe.rs:23` and `Warmup::start_gpu()` already do, and uses a fixed `Rgba8UnormSrgb` format. `encode_cell_pass` and `encode_post_pass` already take any `&TextureView`. 2. **Pixel readback.** Add `Renderer::read_frame(&mut self, out: &mut Vec<u8>)`. It returns tightly packed RGBA of the *post-processed* frame through a reusable, row-padded staging buffer. `capture_last_frame()` (`capture.rs:38`) already reads back an offscreen texture, but it returns only a histogram, skips the post pass, and allocates on every call. It could become a thin wrapper over `read_frame`. 3. **Pinned fonts.** Let `Warmup` or `RendererConfig` accept a preloaded `FontStack` (`FontStack::new`, `felis-shaping/src/lib.rs:354`), so the caller controls exactly which font files are used. ## Scope Only "draw into memory and hand back pixels" belongs in felis. Video encoding, frame timing and the `.fcast` format stay in felis-reel. The smoke test and any screenshot-based golden tests could also use the new API. ## Rejected alternatives (from the felis-reel side) - **A separate CPU rasterizer on felis-shaping.** It would duplicate glyph placement, decorations, cursor shapes, Kitty image placement, reverse video and the post shader, and its output would drift from the real window. - **Screen-capturing felis-client attached to `felis-fcast serve`.** This needs a display, runs only in real time, and does not produce deterministic output.

natsukium commented

2026-09-25 21:39:16 +09:00

Author

Owner

Checked the proposal against current main. The direction holds, but several details in the body are wrong or under-scoped. Plan: split into two PRs; only the first blocks the felis-reel PoC.

PR A: offscreen target and readback

Split size and format out of SurfaceConfiguration first. Renderer.config is also the width/height/format store for resize, reconfigure_surface, capture.rs and the viewport math, so a Target::{Surface, Offscreen} enum cannot be added without this split.
Renderer::new_offscreen(width, height, cfg, warmup): adapter picked with compatible_surface: None, fixed Rgba8UnormSrgb, and background_opacity ignored (so bg_premultiplied is false and the cell pass writes straight RGB).
No post stage offscreen. The post stage reads the host clock (Instant::now() in post.rs), and the bundled trail shader paints the whole frame while TrailState is at its all-zero default. An offscreen caller would have to drive both, so offscreen does not support post shaders at all: new_offscreen returns an error when cfg names a post shader, and set_post_shader(Some(_)) returns a PostShaderError on an offscreen target.
resize works offscreen: it recreates the target texture and the staging buffer. A consumer that needs a fixed output size renders at the exact grid size and pads on its own side, because letterbox_origin centers a grid once the target has a cell of spare space.
read_frame(&mut self, out: &mut Vec<u8>) copies the offscreen texture through a reusable, row-padded staging buffer into tightly packed RGBA. It makes no promise about alpha: DECSCNM and SGR 7 can swap a translucent theme color into a cell background, so a consumer that needs opaque output overwrites alpha itself.
Correction: capture_last_frame cannot become a thin wrapper over read_frame. The smoke test runs on a window surface whose swapchain texture has no COPY_SRC, so the re-encode path stays for the Surface target.

PR B: pinned fonts

Correction: FontStack::new(primary) drops the fallbacks and the bold/italic faces. Instead: (a) a public discovery entry point that takes a caller-built fontdb::Database (the private auto_discover_in already has that shape), and (b) a construction path that accepts the finished FontStack, since the renderer otherwise only gets its stack from auto_discover or the private warmup state.
Golden tests select lavapipe through the environment and assert the adapter name from get_info(). compatible_surface: None with HighPerformance still prefers a hardware adapter when one is present.

Justification

Correction: docs/explanation/rendering/pipeline.md:262 is a "Revisit if" about NextGridFrame pacing for IPC consumers, not about the renderer, so it is not evidence for this change. What felis itself gains is window-less screenshot golden tests (lavapipe in CI) on the same API.

Checked the proposal against current `main`. The direction holds, but several details in the body are wrong or under-scoped. Plan: split into two PRs; only the first blocks the felis-reel PoC. ## PR A: offscreen target and readback - **Split size and format out of `SurfaceConfiguration` first.** `Renderer.config` is also the width/height/format store for `resize`, `reconfigure_surface`, `capture.rs` and the viewport math, so a `Target::{Surface, Offscreen}` enum cannot be added without this split. - **`Renderer::new_offscreen(width, height, cfg, warmup)`**: adapter picked with `compatible_surface: None`, fixed `Rgba8UnormSrgb`, and `background_opacity` ignored (so `bg_premultiplied` is false and the cell pass writes straight RGB). - **No post stage offscreen.** The post stage reads the host clock (`Instant::now()` in `post.rs`), and the bundled trail shader paints the whole frame while `TrailState` is at its all-zero default. An offscreen caller would have to drive both, so offscreen does not support post shaders at all: `new_offscreen` returns an error when `cfg` names a post shader, and `set_post_shader(Some(_))` returns a `PostShaderError` on an offscreen target. - **`resize` works offscreen**: it recreates the target texture and the staging buffer. A consumer that needs a fixed output size renders at the exact grid size and pads on its own side, because `letterbox_origin` centers a grid once the target has a cell of spare space. - **`read_frame(&mut self, out: &mut Vec<u8>)`** copies the offscreen texture through a reusable, row-padded staging buffer into tightly packed RGBA. It makes no promise about alpha: DECSCNM and SGR 7 can swap a translucent theme color into a cell background, so a consumer that needs opaque output overwrites alpha itself. - **Correction:** `capture_last_frame` cannot become a thin wrapper over `read_frame`. The smoke test runs on a window surface whose swapchain texture has no `COPY_SRC`, so the re-encode path stays for the Surface target. ## PR B: pinned fonts - **Correction:** `FontStack::new(primary)` drops the fallbacks and the bold/italic faces. Instead: (a) a public discovery entry point that takes a caller-built `fontdb::Database` (the private `auto_discover_in` already has that shape), and (b) a construction path that accepts the finished `FontStack`, since the renderer otherwise only gets its stack from `auto_discover` or the private warmup state. - Golden tests select lavapipe through the environment and assert the adapter name from `get_info()`. `compatible_surface: None` with `HighPerformance` still prefers a hardware adapter when one is present. ## Justification **Correction:** `docs/explanation/rendering/pipeline.md:262` is a "Revisit if" about `NextGridFrame` pacing for IPC consumers, not about the renderer, so it is not evidence for this change. What felis itself gains is window-less screenshot golden tests (lavapipe in CI) on the same API.

natsukium referenced this issue

2026-09-25 22:31:54 +09:00

render: draw into an offscreen texture and read frames back #465

natsukium referenced this issue from a commit

2026-09-25 23:18:26 +09:00

render: draw into an offscreen texture and read frames back (#465)

natsukium referenced this issue from a pull request that will close it,

2026-09-26 00:31:18 +09:00

render, shaping: let a caller pin the font files a frame is drawn with #466

natsukium closed this issue

2026-09-26 00:45:07 +09:00

natsukium referenced this issue from a commit