Skip to content

ADR 0013: Clipper looks at keyframes ​

  • Status: Accepted
  • Date: 2026-11-21
  • Spec version: 1.3.0
  • Builds on: ADR 0011

Context ​

ADR 0011 scores moments from audio, transcript and motion numbers, and the server's model only reads text. On gameplay recordings with little dialogue this picks silent action badly, and titles describe timestamps rather than what is on screen. Opus Clip-style tools look at the picture. We need the model to see the picture without sending video, which ADR 0011 forbids.

Decision ​

  • Still frames, never video. For the top candidates (at most 24 per video), the Clipper cuts four small JPEG stills (384 px wide, quality 5) per candidate on the editor's machine and sends them with the score request. Each candidate's window is cropped of black bars first. Stills are the only pixels that leave the machine, and only online.
  • Limits. At most 4 stills per candidate and 96 per request. A still over 60 KiB encoded is dropped. The score request body is capped at 8 MiB. Stills are billed as 200 tokens each in the usage meter.
  • Model output. For a candidate with stills the model returns a one-sentence visual summary (at most 140 characters) and up to 4 on-screen moments with times inside the window and labels of at most 40 characters. Moments outside the window are dropped. The summary feeds the title and the marker labels.
  • Capability, not assumption. The server advertises features.vision in the handshake only when the configured model accepts images. Without it, the Clipper sends no stills and scores from text only, as ADR 0011 does.
  • Offline is unchanged. The offline engine never extracts or sends frames.
  • Settings. ClipSettings.visual (default on) turns the stills off for a run.

Consequences ​

  • The Clipper does extra local work (about 7 short ffmpeg decodes per candidate, roughly two minutes for 24 candidates of a 20-minute recording on a loaded machine) and uploads up to about 6 MB of JPEG per request.
  • The server must hold an image-capable model. The research provider already does; the Claude bridge forwards images to the Claude CLI and reads them from a temporary directory it deletes afterwards.
  • Scores with stills and scores without them are not directly comparable.

Revisit when ​

Pixels leave the machine in any other form (video, full-resolution frames), a model that cannot take images becomes the only one configured, or stills cost more than the clips they improve.

Internal documentation. Not for clients.