Appearance
Clipper app
The Clipper app is a single executable editors run on their own computer. It turns long raw video (streams, gameplay, interviews) into short vertical clips with captions, titles and hashtags, filed and ready to upload. It works with no internet and works better connected to this web app. The decision is ADR 0011; the server half is in Clipper server.
Code: packages/clipper (the engine), apps/clipper (command line, local studio, build).
How a video becomes clips
flowchart LR
V[Video file or link] --> A[Audio + picture analysis]
V --> T[Transcript with word times]
A --> D[Detect moments]
T --> D
D --> S[Judge and rank]
S --> C[Titles, hashtags, opener]
C --> R[Render 9:16 with captions]
R --> O[File by category + upload sheet]
| Stage | What it does | Offline | Online (connected) |
|---|---|---|---|
| Audio | 16 kHz audio; loudness, rises above the recent baseline, brightness per half second | this computer | this computer |
| Picture | one 4 fps low-resolution pass: motion, where it moves, scene cuts | this computer | this computer |
| Transcript | words with times; voice-activity detection first so music and game sound are not "heard" as speech | whisper.cpp | compressed audio (Opus, about 3.5 MB per 20 minutes) goes to the server |
| Detect | fuses loudness surprise, motion, cuts and speech excitement into one series; peaks grow into moments with a lead-in and payoff; edges snap to sentence, pause or shot boundaries; overlaps removed | this computer | this computer |
| Judge | scores each moment 0 to 100 | the signal score | the server's model sees the transcript and the numbers, blended 70/30 with the signal score |
| Copy | title, description, hashtags per platform, on-screen opener | templates built from the clip's own words | the server's model |
| Render | fit, fill or focus to 1080x1920, word-by-word captions, loudness -14 LUFS, H.264/AAC; thumbnail from a detailed frame near the peak | this computer | this computer |
Every online step that fails falls back to its offline twin and the project says so in its warnings. Nothing is skipped silently.
What leaves the computer
Only when connected and "work offline" is off: compressed audio for transcription, transcript text and numeric signals for scoring and copy, and project and clip metadata (names, times, scores, titles, status). When the server's model accepts images, it also gets up to four small JPEG stills (384 px wide) per candidate, cut on this computer (ADR 0013). The video, the rendered clips and file paths never leave.
Looking at the picture
Online, the app takes the top 24 candidate moments, cuts four still frames from each, and sends them with the scores. Black bars are cropped from each window first. The model describes what is on screen and names up to four moments inside the window: the description informs the clip's title, and the moments become labelled markers. The studio's Look at the picture checkbox, or --no-vision on the command line, turns it off. Offline runs never look at the picture, and a model that cannot take images scores from sound and text only.
Upload-ready copy
Titles, descriptions, hashtags and the on-screen opener never contain profanity (the server's prompts forbid it and the offline copy skips such lines). Burned captions and .srt files mask profanity (s***) unless you pass --no-censor; the transcript itself is unchanged.
Layouts
- split: a streamer's webcam on top and the content below, in one 9:16 frame. Used when a small face stays in a corner of the picture for most of the clip; the webcam size comes from its face size, so the inset is approximate. The webcam panel keeps the inset's own shape, and the content crop uses the side of the picture away from the inset. Without a webcam, a split clip renders as fill.
- fill: a full-bleed 9:16 crop that follows the people on screen, and follows the action where no face is found. Used for automatic choices without a webcam, including gameplay and cutscenes. See Subject tracking. Dark footage is lifted toward a normal brightness. When the game burns its own subtitle lines into the bottom quarter of the picture (a line in most sampled frames), that band is cut off before the crop, so only our captions remain; a clip without such lines is not cut. The cut shortens the picture, so the crop is upscaled more and reads a little softer.
- fit: the whole frame over a blurred copy of itself. Only when asked for (
--layout fit): the picture stays small in the middle of the frame. - focus: a square crop around the motion over a blurred copy. Only when asked for (
--layout focus).
Automatic choice (auto) picks split when a webcam is found, otherwise fill.
Black bars (letterbox) are measured on each clip's own picture, because gameplay is full frame while cutscenes are letterboxed, and removed before the layout is applied.
Subject tracking
Fill and split crops follow the people on screen. Two embedded models run on the picture: YuNet (MIT, 230 KB) finds faces six times a second and links them into tracks, and NanoDet (Apache-2.0, 3.8 MB) finds people every third analysis frame. A face track counts only when its face sits in the upper part of a detected person for most of its frames, or when it stays on screen for three seconds with a high score; this drops hands and forearms that look like faces. The active speaker is picked from mouth movement and speech timing, and the crop holds that person. Two people close together share the crop when both fit. Two people farther apart than about 430 px in the source cannot both fit a 9:16 window, so the crop holds the speaker. The crop centre sits so the eyes are near the top third of the frame, and it zooms in (up to 2x) when faces are small in the picture. The zoom is set once for the clip. Where no face is found, the crop follows a person's head, then motion; the clip gets a warning when that fallback covers more than half of it.
Each clip records its framing in the project (framing: source, layout, zoom, coverage, fallback seconds). On one thread under load, analysis took about 100 s for a 60 s clip of 1080p footage.
Output
<output>/<name>-<id>/
clips/<category>/NN-title.mp4 the clip (1080x1920, 30 fps, or 60 fps for sources of 50 fps and up; H.264 CRF 18; AAC 192 kbps; -14 LUFS)
clips/<category>/NN-title.jpg thumbnail from the peak
clips/<category>/NN-title.srt subtitles
clips/<category>/NN-title.json score, reason, copy
manifest.json the whole project
upload.json, upload-queue.csv title, description, hashtags per platform
index.html contact sheet
project.json editable state (the studio reads and writes this)
work/ caches: audio, signals, transcript (safe to delete)Categories: action, funny, reaction, story, insight, scenery, other. Re-running a project reuses work/ when the file has not changed.
Unprocessed clips
--output processed|unprocessed|both (the studio's Output setting; default processed) chooses what a project makes. processed is the finished 9:16 video above. unprocessed makes raw cuts for the video editor: each moment is cut from the source at the source's own size and frame rate, with handles (--handle <seconds>, default 1.5, at most 10), with no crop, captions or loudness change. both makes the two.
<output>/<name>-<id>/
clips-unprocessed/<importance>/NN_IMPORTANCE_category_label_hh-mm-ss.mp4 the raw cut
clips-unprocessed/<importance>/....review.mp4 scan copy: 480 tall, 30 fps, source timecode, HANDLE flag outside the clip, label, timeline bar with a tick per marker, marker banners
clips-unprocessed/<importance>/....json label, importance, score, reason, source time, handles, markers
clips-unprocessed/<importance>/....markers.csv markers: file seconds, source timecode, kind, strength, label
clips-unprocessed/<importance>/....srt subtitles timed to the cut (follows the censor setting)
editor-labels.csv one row per clip
markers.edl CMX 3600 edit list: one event per clip, markers as yellow locators
unprocessed.html the labelled list, scan copies and markers, sorted by importanceImportance is high, medium or low: the top third of clips by score is high, the middle third medium, the rest low. A clip is raised one level when the picture or the sound shows at least three strong moments, or the model saw something in the picture.
Markers are up to six moments per clip: picture movement, sound rises, the key spoken sentence (line), scene cuts, the strongest moment (peak), and what the model saw in the picture when it was online (visual). Their times are seconds from the start of the clip; in the raw file they are offset by the handle.
Labels are masked for profanity, as in the processed output. Raw cuts are re-encoded (H.264, CRF 16) so the cut points are exact; they are not stream copies. Raw-only projects have no frame board. Send raw clips with clipper handoff <folder> --video <id> --variant raw (footage videos default to raw, edit videos to processed); the studio's Send dialog offers the same choice.
The EDL follows the CMX 3600 format with 01:00:00:00 as the record start. It has been checked against the format, not yet opened in an editor.
Brand templates
A template sets how captions and the hook look, and can add a logo, an intro and an outro to every clip. classic is the default and matches the look clips had before templates existed. The built-in templates are classic, clean and bold.
A template file is JSON with "schemaVersion": 1. Unknown keys are an error.
name(required, lower-case letters, digits and dashes) anddescriptioncaptions:font,fontFile(a .ttf or .otf on this computer, installed for the captions),size,textColour,activeColour(the word being spoken; equal totextColourturns the highlight off),outlineColour,outlineWidth,box(a solid panel behind the text instead of the outline:colour,opacity,padding),position(low,middleorhigh),uppercase,pophook:size,textColour,outlineColour,outlineWidth,boxlogo:file(a PNG),corner,widthPx,opacity,marginPxintroandoutro:{ "file": "clip.mp4" }, joined before and after each clip
Colours are six-digit hex, such as #FFE000. Relative paths resolve from the template file's folder. Choose a template with --template <name|file>, or in the studio's New project form under Caption look.
Limits: a logo, intro or outro is added to the rendered file after the frame board is built, and an intro or outro lengthens the clip. Markers, the edit list and handoff times refer to the clip body, not to the branded file. If a template cannot be applied, the clip is kept without it and the project gets a warning.
Agents
Scripts and agents can drive the CLI without the studio:
clipper clip <file|link> --jsonprints one JSON object:projectDir,template,warnings, andclips. Each clip hasrank,id,category,start,end,score,hook,fileandplatformScores(per platform:scoreandreasons, the rules that moved it). Progress is on stderr.clipper projects --jsonprints the project list as JSON.clipper template list [--json],template show <name|file>andtemplate validate <file>work without starting the runtime.validateexits 1 and lists the problems when a file is wrong.
The skill for coding agents is in apps/clipper/skills/shootgrow-clipper/SKILL.md.
Install and connect
- Download the executable for your system from the Clipper page in the web app (Owner, Crew), or from
GET /clipper/v1/download/<platform>. clipper doctorshows what is missing. ffmpeg with libx264 and libass is required.clipper setupdownloads the transcription model and the voice-activity model (and whisper.cpp on Windows). On Linux buildwhisper-clifrom whisper.cpp; on macOSbrew install whisper-cpp.- In the web app create an API key (Clipper page, "New key"). The key is shown once.
clipper connect https://<your-server> sgc_...or use the Connect dialog in the studio.clipper statusshows the user, the key, which server steps are on and this month's usage.clipperopens the studio athttp://127.0.0.1:4317.clipper clip <file|link>runs without it.
The key is stored in the app's data folder with owner-only permissions. clipper disconnect removes it.
Commands
| Command | |
|---|---|
clipper studio [--port 4317] [--no-open] | The studio in the browser: new project, live progress, library, review grid, trim and re-render, approve, export |
clipper clip <file|link> [options] | One project from the terminal; the options are listed below the table |
clipper connect <server> <key> / disconnect / status | The server link |
clipper projects / reprocess <folder> / sync / doctor / setup [--model base|small] | Housekeeping; reprocess runs a project again from its saved analysis |
clipper videos / handoff <folder> --video <id> [--clips id,id] [--variant raw|processed] | Send clips to a video as a crew upload (see below). Footage videos default to raw, edit videos to processed |
clipper template list / show <name|file> / validate <file> | Brand templates: the built-in looks, one as JSON, or the problems with a template file |
clipper clip options (the same settings as the studio's New project form):
--offline: do everything on this computer, even when connected--mode auto|gaming|talk--min <s> --max <s>: clip length range in seconds (default 15 to 60)--keep <n>: number of clips (default depends on the length)--min-score <0-100>: drop clips scoring lower (default 30)--layout auto|fit|fill|focus|split--track auto|motion:auto(default) follows faces and people onfillandsplitclips and falls back to motion when none is found;motionskips face tracking--no-captions: no burned captions--no-censor: do not mask profanity in captions (masked by default)--no-vision: online, do not send small still frames of candidate moments to the server--lang <code>: spoken language (default: detect)--name <text>: project name--out <folder>: where to write projects--platforms youtube_shorts,tiktok,instagram_reels--output processed|unprocessed|both: see Unprocessed clips--handle <s>: seconds of footage kept before and after each unprocessed clip (default 1.5, 0 to 10)--template classic|clean|bold|<file>: the caption look, hook and any logo, intro or outro (see Brand templates); defaultclassic--json: print the result as one JSON object on stdout; progress stays on stderr (see Agents)
Environment overrides: CLIPPER_HOME (data folder), CLIPPER_OUTPUT, CLIPPER_SERVER, CLIPPER_API_KEY, CLIPPER_FFMPEG, CLIPPER_FFPROBE, CLIPPER_WHISPER, CLIPPER_WHISPER_MODEL, CLIPPER_VAD_MODEL, CLIPPER_YTDLP.
Sending clips to production
clipper videos lists the videos your key can send to, with what each one takes (footage or the edit) and why a video is closed. Videographers send footage; editors send the edit. Owners and managers see the list but cannot send.
clipper handoff <folder> --video <id> sends the approved clips of a project (or the clips you name with --clips; with no approval, every rendered clip). It starts with a check: each clip must be rendered, or nothing is sent. Then it uploads the clips one at a time, in order, to the video's next version, using the same upload the web app uses. Each sent clip records its video and version on the project, and an audit event handoff.clip is queued. If one clip fails, the rest are not sent and the message says how many were sent.
Logs, metrics, audit
The app keeps its own log, metrics and audit events on disk (spool.ndjson) and sends them to the server whenever it is connected: project created, stage timings, render counts, fallbacks, edits, approvals, exports, connects. Offline events wait on disk and are delivered at the next connection. Events never contain file paths or transcript text. The owner reads them under Audit in the web app.
Building
bun run --cwd apps/clipper package [linux|windows|macos-arm64|macos-x64] writes single-file executables and dist/releases.json (size and SHA-256). Upload them to the FILES bucket as clipper/releases/<platform>/clipper (.exe for Windows) so /clipper/v1/download/<platform> serves them, and set the clipper.latestVersion setting.
Known limits
- Windows and macOS executables are cross-compiled but only the Linux one has been run.
- Subject tracking covers the
fillandsplitlayouts. Gameplay with no faces or people on screen (third-person views with the character small, cutscenes with faces turned away) still follows motion;focusandfitstay on motion. - Hands and forearms are filtered by the person check, but one that no person box confirms can still be taken for a face and hold the crop for a second or so.
- Two people further apart than about 430 px in the source cannot both fit a 9:16 window. The crop then holds the speaker and may cut the other person at the edge.
- Zoom upscales the crop: a 508 px wide crop from a 1920 px source becomes 1080 px, so the picture is softer than a full-frame crop.
- The webcam inset is estimated from its face size. A webcam at another position or with a different shape may be missed or mis-sized.
- Framing is saved per clip in the project. A warning says when face tracking failed or fell back to motion for over half of a clip.
- Dubbing, translation, direct publishing and scheduled posting are not built. The upload sheet is the hand-off.
- Rendering uses the CPU (libx264). A slow computer renders slowly.
Frame board
Every rendered clip gets a frame board: ten frames spread across the clip, stamped with the time of each frame, on one contact sheet.
- Files:
clips/<category>/NN-title.board.jpg(the sheet: 5 by 2 tiles, 1080 by 768) andclips/<category>/NN-title.frames/fNN.jpg(the ten frames, 540 by 960).frameboard.htmlin the project folder shows every clip's board, andindex.htmllinks to it. Paths are listed inmanifest.jsonunder each clip'sboard. - Frame choice: ten targets spread evenly from 0.5 s in to 0.5 s before the end. Each moves up to 1 s to the best nearby sample; black and flat frames are skipped.
- Cost: each clip takes two extra ffmpeg runs, a small grey decode to choose frames and one run that writes the sheet and frames. Re-rendering a clip replaces its board.
- Studio: each card shows a strip of its ten frames under the video; click one to jump there. Frame board opens the sheet with a frame picker (arrow keys move, Home and End jump). Copy path copies the full path of the sheet. The Frame boards view at the top of a project shows every clip's sheet.
Send to production
Send to production on a card, or Send approved to production in the project header, uploads the rendered clip file to a video in the production app as a new version. It runs in the background and the job panel shows its progress. The server decides from the key owner's role whether the clips go to the videographer as footage or to the editor as the edit. Each sent clip shows "Sent to <video> v<N>" on its card.
The send needs a connected key. Only videos the key can upload to can be chosen; the others show the reason. If Clipper is offline or not connected, the dialog says so and offers Connect.