Skip to content

Clipper app ​

The Clipper app is a single executable editors run on their own computer. It turns long raw video (streams, gameplay, interviews) into short vertical clips with captions, titles and hashtags, filed and ready to upload. It works with no internet and works better connected to this web app. The decision is ADR 0011; the server half is in Clipper server.

Code: packages/clipper (the engine), apps/clipper (command line, local studio, build).

How a video becomes clips ​

flowchart LR
  V[Video file or link] --> A[Audio + picture analysis]
  V --> T[Transcript with word times]
  A --> D[Detect moments]
  T --> D
  D --> S[Judge and rank]
  S --> C[Titles, hashtags, opener]
  C --> R[Render 9:16 with captions]
  R --> O[File by category + upload sheet]
StageWhat it doesOfflineOnline (connected)
Audio16 kHz audio; loudness, rises above the recent baseline, brightness per half secondthis computerthis computer
Pictureone 4 fps low-resolution pass: motion, where it moves, scene cutsthis computerthis computer
Transcriptwords with times; voice-activity detection first so music and game sound are not "heard" as speechwhisper.cppcompressed audio (Opus, about 3.5 MB per 20 minutes) goes to the server
Detectfuses loudness surprise, motion, cuts and speech excitement into one series; peaks grow into moments with a lead-in and payoff; edges snap to sentence, pause or shot boundaries; overlaps removedthis computerthis computer
Judgescores each moment 0 to 100the signal scorethe server's model sees the transcript and the numbers, blended 70/30 with the signal score
Copytitle, description, hashtags per platform, on-screen openertemplates built from the clip's own wordsthe server's model
Renderfit, fill or focus to 1080x1920, word-by-word captions, loudness -14 LUFS, H.264/AAC; thumbnail from a detailed frame near the peakthis computerthis computer

Every online step that fails falls back to its offline twin and the project says so in its warnings. Nothing is skipped silently.

What leaves the computer ​

Only when connected and "work offline" is off: compressed audio for transcription, transcript text and numeric signals for scoring and copy, and project and clip metadata (names, times, scores, titles, status). When the server's model accepts images, it also gets up to four small JPEG stills (384 px wide) per candidate, cut on this computer (ADR 0013). The video, the rendered clips and file paths never leave.

Looking at the picture ​

Online, the app takes the top 24 candidate moments, cuts four still frames from each, and sends them with the scores. Black bars are cropped from each window first. The model describes what is on screen and names up to four moments inside the window: the description informs the clip's title, and the moments become labelled markers. The studio's Look at the picture checkbox, or --no-vision on the command line, turns it off. Offline runs never look at the picture, and a model that cannot take images scores from sound and text only.

Upload-ready copy ​

Titles, descriptions, hashtags and the on-screen opener never contain profanity (the server's prompts forbid it and the offline copy skips such lines). Burned captions and .srt files mask profanity (s***) unless you pass --no-censor; the transcript itself is unchanged.

Layouts ​

  • split: a streamer's webcam on top and the content below, in one 9:16 frame. Used when a small face stays in a corner of the picture for most of the clip; the webcam size comes from its face size, so the inset is approximate. The webcam panel keeps the inset's own shape, and the content crop uses the side of the picture away from the inset. Without a webcam, a split clip renders as fill.
  • fill: a full-bleed 9:16 crop that follows the people on screen, and follows the action where no face is found. Used for automatic choices without a webcam, including gameplay and cutscenes. See Subject tracking. Dark footage is lifted toward a normal brightness. When the game burns its own subtitle lines into the bottom quarter of the picture (a line in most sampled frames), that band is cut off before the crop, so only our captions remain; a clip without such lines is not cut. The cut shortens the picture, so the crop is upscaled more and reads a little softer.
  • fit: the whole frame over a blurred copy of itself. Only when asked for (--layout fit): the picture stays small in the middle of the frame.
  • focus: a square crop around the motion over a blurred copy. Only when asked for (--layout focus).

Automatic choice (auto) picks split when a webcam is found, otherwise fill.

Black bars (letterbox) are measured on each clip's own picture, because gameplay is full frame while cutscenes are letterboxed, and removed before the layout is applied.

Subject tracking ​

Fill and split crops follow the people on screen. Two embedded models run on the picture: YuNet (MIT, 230 KB) finds faces six times a second and links them into tracks, and NanoDet (Apache-2.0, 3.8 MB) finds people every third analysis frame. A face track counts only when its face sits in the upper part of a detected person for most of its frames, or when it stays on screen for three seconds with a high score; this drops hands and forearms that look like faces. The active speaker is picked from mouth movement and speech timing, and the crop holds that person. Two people close together share the crop when both fit. Two people farther apart than about 430 px in the source cannot both fit a 9:16 window, so the crop holds the speaker. The crop centre sits so the eyes are near the top third of the frame, and it zooms in (up to 2x) when faces are small in the picture. The zoom is set once for the clip. Where no face is found, the crop follows a person's head, then motion; the clip gets a warning when that fallback covers more than half of it.

Each clip records its framing in the project (framing: source, layout, zoom, coverage, fallback seconds). On one thread under load, analysis took about 100 s for a 60 s clip of 1080p footage.

Output ​

<output>/<name>-<id>/
  clips/<category>/NN-title.mp4   the clip (1080x1920, 30 fps, or 60 fps for sources of 50 fps and up; H.264 CRF 18; AAC 192 kbps; -14 LUFS)
  clips/<category>/NN-title.jpg   thumbnail from the peak
  clips/<category>/NN-title.srt   subtitles
  clips/<category>/NN-title.json  score, reason, copy
  manifest.json                   the whole project
  upload.json, upload-queue.csv   title, description, hashtags per platform
  index.html                      contact sheet
  project.json                    editable state (the studio reads and writes this)
  work/                           caches: audio, signals, transcript (safe to delete)

Categories: action, funny, reaction, story, insight, scenery, other. Re-running a project reuses work/ when the file has not changed.

Unprocessed clips ​

--output processed|unprocessed|both (the studio's Output setting; default processed) chooses what a project makes. processed is the finished 9:16 video above. unprocessed makes raw cuts for the video editor: each moment is cut from the source at the source's own size and frame rate, with handles (--handle <seconds>, default 1.5, at most 10), with no crop, captions or loudness change. both makes the two.

<output>/<name>-<id>/
  clips-unprocessed/<importance>/NN_IMPORTANCE_category_label_hh-mm-ss.mp4        the raw cut
  clips-unprocessed/<importance>/....review.mp4   scan copy: 480 tall, 30 fps, source timecode, HANDLE flag outside the clip, label, timeline bar with a tick per marker, marker banners
  clips-unprocessed/<importance>/....json         label, importance, score, reason, source time, handles, markers
  clips-unprocessed/<importance>/....markers.csv  markers: file seconds, source timecode, kind, strength, label
  clips-unprocessed/<importance>/....srt          subtitles timed to the cut (follows the censor setting)
  editor-labels.csv                               one row per clip
  markers.edl                                     CMX 3600 edit list: one event per clip, markers as yellow locators
  unprocessed.html                                the labelled list, scan copies and markers, sorted by importance

Importance is high, medium or low: the top third of clips by score is high, the middle third medium, the rest low. A clip is raised one level when the picture or the sound shows at least three strong moments, or the model saw something in the picture.

Markers are up to six moments per clip: picture movement, sound rises, the key spoken sentence (line), scene cuts, the strongest moment (peak), and what the model saw in the picture when it was online (visual). Their times are seconds from the start of the clip; in the raw file they are offset by the handle.

Labels are masked for profanity, as in the processed output. Raw cuts are re-encoded (H.264, CRF 16) so the cut points are exact; they are not stream copies. Raw-only projects have no frame board. Send raw clips with clipper handoff <folder> --video <id> --variant raw (footage videos default to raw, edit videos to processed); the studio's Send dialog offers the same choice.

The EDL follows the CMX 3600 format with 01:00:00:00 as the record start. It has been checked against the format, not yet opened in an editor.

Brand templates ​

A template sets how captions and the hook look, and can add a logo, an intro and an outro to every clip. classic is the default and matches the look clips had before templates existed. The built-in templates are classic, clean and bold.

A template file is JSON with "schemaVersion": 1. Unknown keys are an error.

  • name (required, lower-case letters, digits and dashes) and description
  • captions: font, fontFile (a .ttf or .otf on this computer, installed for the captions), size, textColour, activeColour (the word being spoken; equal to textColour turns the highlight off), outlineColour, outlineWidth, box (a solid panel behind the text instead of the outline: colour, opacity, padding), position (low, middle or high), uppercase, pop
  • hook: size, textColour, outlineColour, outlineWidth, box
  • logo: file (a PNG), corner, widthPx, opacity, marginPx
  • intro and outro: { "file": "clip.mp4" }, joined before and after each clip

Colours are six-digit hex, such as #FFE000. Relative paths resolve from the template file's folder. Choose a template with --template <name|file>, or in the studio's New project form under Caption look.

Limits: a logo, intro or outro is added to the rendered file after the frame board is built, and an intro or outro lengthens the clip. Markers, the edit list and handoff times refer to the clip body, not to the branded file. If a template cannot be applied, the clip is kept without it and the project gets a warning.

Agents ​

Scripts and agents can drive the CLI without the studio:

  • clipper clip <file|link> --json prints one JSON object: projectDir, template, warnings, and clips. Each clip has rank, id, category, start, end, score, hook, file and platformScores (per platform: score and reasons, the rules that moved it). Progress is on stderr.
  • clipper projects --json prints the project list as JSON.
  • clipper template list [--json], template show <name|file> and template validate <file> work without starting the runtime. validate exits 1 and lists the problems when a file is wrong.

The skill for coding agents is in apps/clipper/skills/shootgrow-clipper/SKILL.md.

Install and connect ​

  1. Download the executable for your system from the Clipper page in the web app (Owner, Crew), or from GET /clipper/v1/download/<platform>.
  2. clipper doctor shows what is missing. ffmpeg with libx264 and libass is required. clipper setup downloads the transcription model and the voice-activity model (and whisper.cpp on Windows). On Linux build whisper-cli from whisper.cpp; on macOS brew install whisper-cpp.
  3. In the web app create an API key (Clipper page, "New key"). The key is shown once.
  4. clipper connect https://<your-server> sgc_... or use the Connect dialog in the studio. clipper status shows the user, the key, which server steps are on and this month's usage.
  5. clipper opens the studio at http://127.0.0.1:4317. clipper clip <file|link> runs without it.

The key is stored in the app's data folder with owner-only permissions. clipper disconnect removes it.

Commands ​

Command
clipper studio [--port 4317] [--no-open]The studio in the browser: new project, live progress, library, review grid, trim and re-render, approve, export
clipper clip <file|link> [options]One project from the terminal; the options are listed below the table
clipper connect <server> <key> / disconnect / statusThe server link
clipper projects / reprocess <folder> / sync / doctor / setup [--model base|small]Housekeeping; reprocess runs a project again from its saved analysis
clipper videos / handoff <folder> --video <id> [--clips id,id] [--variant raw|processed]Send clips to a video as a crew upload (see below). Footage videos default to raw, edit videos to processed
clipper template list / show <name|file> / validate <file>Brand templates: the built-in looks, one as JSON, or the problems with a template file

clipper clip options (the same settings as the studio's New project form):

  • --offline: do everything on this computer, even when connected
  • --mode auto|gaming|talk
  • --min <s> --max <s>: clip length range in seconds (default 15 to 60)
  • --keep <n>: number of clips (default depends on the length)
  • --min-score <0-100>: drop clips scoring lower (default 30)
  • --layout auto|fit|fill|focus|split
  • --track auto|motion: auto (default) follows faces and people on fill and split clips and falls back to motion when none is found; motion skips face tracking
  • --no-captions: no burned captions
  • --no-censor: do not mask profanity in captions (masked by default)
  • --no-vision: online, do not send small still frames of candidate moments to the server
  • --lang <code>: spoken language (default: detect)
  • --name <text>: project name
  • --out <folder>: where to write projects
  • --platforms youtube_shorts,tiktok,instagram_reels
  • --output processed|unprocessed|both: see Unprocessed clips
  • --handle <s>: seconds of footage kept before and after each unprocessed clip (default 1.5, 0 to 10)
  • --template classic|clean|bold|<file>: the caption look, hook and any logo, intro or outro (see Brand templates); default classic
  • --json: print the result as one JSON object on stdout; progress stays on stderr (see Agents)

Environment overrides: CLIPPER_HOME (data folder), CLIPPER_OUTPUT, CLIPPER_SERVER, CLIPPER_API_KEY, CLIPPER_FFMPEG, CLIPPER_FFPROBE, CLIPPER_WHISPER, CLIPPER_WHISPER_MODEL, CLIPPER_VAD_MODEL, CLIPPER_YTDLP.

Sending clips to production ​

clipper videos lists the videos your key can send to, with what each one takes (footage or the edit) and why a video is closed. Videographers send footage; editors send the edit. Owners and managers see the list but cannot send.

clipper handoff <folder> --video <id> sends the approved clips of a project (or the clips you name with --clips; with no approval, every rendered clip). It starts with a check: each clip must be rendered, or nothing is sent. Then it uploads the clips one at a time, in order, to the video's next version, using the same upload the web app uses. Each sent clip records its video and version on the project, and an audit event handoff.clip is queued. If one clip fails, the rest are not sent and the message says how many were sent.

Logs, metrics, audit ​

The app keeps its own log, metrics and audit events on disk (spool.ndjson) and sends them to the server whenever it is connected: project created, stage timings, render counts, fallbacks, edits, approvals, exports, connects. Offline events wait on disk and are delivered at the next connection. Events never contain file paths or transcript text. The owner reads them under Audit in the web app.

Building ​

bun run --cwd apps/clipper package [linux|windows|macos-arm64|macos-x64] writes single-file executables and dist/releases.json (size and SHA-256). Upload them to the FILES bucket as clipper/releases/<platform>/clipper (.exe for Windows) so /clipper/v1/download/<platform> serves them, and set the clipper.latestVersion setting.

Known limits ​

  • Windows and macOS executables are cross-compiled but only the Linux one has been run.
  • Subject tracking covers the fill and split layouts. Gameplay with no faces or people on screen (third-person views with the character small, cutscenes with faces turned away) still follows motion; focus and fit stay on motion.
  • Hands and forearms are filtered by the person check, but one that no person box confirms can still be taken for a face and hold the crop for a second or so.
  • Two people further apart than about 430 px in the source cannot both fit a 9:16 window. The crop then holds the speaker and may cut the other person at the edge.
  • Zoom upscales the crop: a 508 px wide crop from a 1920 px source becomes 1080 px, so the picture is softer than a full-frame crop.
  • The webcam inset is estimated from its face size. A webcam at another position or with a different shape may be missed or mis-sized.
  • Framing is saved per clip in the project. A warning says when face tracking failed or fell back to motion for over half of a clip.
  • Dubbing, translation, direct publishing and scheduled posting are not built. The upload sheet is the hand-off.
  • Rendering uses the CPU (libx264). A slow computer renders slowly.

Frame board ​

Every rendered clip gets a frame board: ten frames spread across the clip, stamped with the time of each frame, on one contact sheet.

  • Files: clips/<category>/NN-title.board.jpg (the sheet: 5 by 2 tiles, 1080 by 768) and clips/<category>/NN-title.frames/fNN.jpg (the ten frames, 540 by 960). frameboard.html in the project folder shows every clip's board, and index.html links to it. Paths are listed in manifest.json under each clip's board.
  • Frame choice: ten targets spread evenly from 0.5 s in to 0.5 s before the end. Each moves up to 1 s to the best nearby sample; black and flat frames are skipped.
  • Cost: each clip takes two extra ffmpeg runs, a small grey decode to choose frames and one run that writes the sheet and frames. Re-rendering a clip replaces its board.
  • Studio: each card shows a strip of its ten frames under the video; click one to jump there. Frame board opens the sheet with a frame picker (arrow keys move, Home and End jump). Copy path copies the full path of the sheet. The Frame boards view at the top of a project shows every clip's sheet.

Send to production ​

Send to production on a card, or Send approved to production in the project header, uploads the rendered clip file to a video in the production app as a new version. It runs in the background and the job panel shows its progress. The server decides from the key owner's role whether the clips go to the videographer as footage or to the editor as the edit. Each sent clip shows "Sent to <video> v<N>" on its card.

The send needs a connected key. Only videos the key can upload to can be chosen; the others show the reason. If Clipper is offline or not connected, the dialog says so and offers Connect.

Internal documentation. Not for clients.