Skip to content

Research (the scrape) ​

One pipeline researches a lead (before the call) and a client (when the account exists, and on demand). Both get the same brief, stored in research_job / research_brief, shown by the same panel. Ported from Reroot's client research and lead research briefs; the design is in ADR 0009.

flowchart LR
  A[Run research] --> B[Create job + enqueue research.run]
  B --> C[Render home + up to 7 pages]
  C --> D[Read the code: JSON-LD, tel, mailto, platform, colours]
  D --> E[Render up to 5 social profile pages]
  E --> F[Pass 1: facts, model with web search]
  F --> G{Market empty?}
  G -->|yes| H[Pass 1b: focused market search]
  G -->|no| I
  H --> I[Pass 2: analysis, no tools]
  I --> J[Assemble: page code beats the model]
  J --> K[Verify figures, store brief v2]

What it finds ​

Brief v2 has twelve sections. The workspace page shows each, says how many have data and names the ones with nothing.

SectionSource
Overview: summary, tagline, founded, team, languages, brand coloursSite text and code (theme-color, JSON-LD), then web search
Contact and hours: phones, emails, addresses, hours, booking linkPage code first (tel/mailto links, schema.org JSON-LD), then web search
What they sell: offerings, products with prices, USPs, how customers buySite pages (menu, services, shop, pricing)
Reputation: rating, review count, sentiment, quotes, awards, pressJSON-LD aggregateRating, then web search (Google, Trustpilot, press)
Website: platform, booking / ordering / video, pages, problemsPage code (platform signature, forms, video embeds) and the analyst
Social accounts: handle, bio, followers, posting cadence, themesLinks on the site, profile pages, web search
VideosVideo and reel links; web search
Content and ideas: themes, gaps, video ideas with hooksAnalyst pass
Audience: personasAnalyst pass
Market: competitors, similar businesses, creators, trends, seasonality, price positionWeb search; a focused extra pass if pass 1 found none
Sales prep: recommended package, quick wins, objections, talking pointsAnalyst pass, choosing only from the agency's real packages
Evidence: pages read and failed, profiles read, passes, structured dataThe run itself

Facts read from page code override the model: a phone number in a tel: link beats one the model remembers. Socials and videos found on the site are never dropped by the model.

Never invent figures. A follower or view count is kept only if that number appears in the text of a page this run rendered. Counts the model saw only in its own searches cannot be checked, so they are removed (the link stays). Every URL must be a well-formed public http(s) address; private and loopback hosts are refused, both for rendering and in the output.

Where it runs ​

PieceWhereNotes
Page renderingCloudflare Browser Rendering, the BROWSER binding (Cloudflare.Browser("BROWSER") in alchemy.run.ts)quickAction("content") returns the HTML after scripts ran, so JS-heavy sites are read like a visitor sees them. Images, media, fonts and styles are skipped.
The runThe jobs queue, job type research.run v1 (limit 8 minutes)Not in a request: it takes about a minute. Idempotent: a finished job is left alone, so a retry changes nothing.
ModelInjected: see below
StorageD1History is kept; the newest complete brief is "current" and stays visible during a re-run.

The model backend (dependency injection) ​

The pipeline depends on three small interfaces in packages/api/src/research/ports.ts: PageRenderer, LlmProvider, ResearchStore. The server is the injection root (apps/server/src/services.ts: getResearch); tests pass fakes.

BackendUseHow
claude-cliDevelopment only: the Claude app, signed in as youA Worker cannot spawn a process, so bun run claude:bridge runs a small loopback server that calls your claude CLI with WebSearch and WebFetch only. Same idea as Reroot's ClaudeCliProvider: your own login, no API key.
anthropicProduction: the Claude PlatformMessages API with an API key and the server-side web_search tool.

Selection (select.ts): RESEARCH_LLM = auto (default), claude-cli, anthropic or off. auto prefers the API key, then the bridge, then nothing. Production never uses the bridge: on the production stage it is refused even if configured. With nothing configured, starting research is refused with the exact reason; there is no fake brief.

Development ​

bash
bun run claude:bridge     # prints CLAUDE_BRIDGE_URL and CLAUDE_BRIDGE_TOKEN
# put both lines in apps/server/.env, then:
bun run dev

The bridge binds to 127.0.0.1, requires the token, never gives the CLI Bash/Write/Edit/Read, and runs at most two calls at once. Set CLAUDE_BRIDGE_MODEL to change the model (default sonnet).

Production ​

Set ANTHROPIC_API_KEY (and optionally ANTHROPIC_MODEL, default claude-sonnet-4-5) as deployment secrets. A secret is only bound to the Worker when it is set at deploy time, so a deploy without a key still works and research reports "not configured". Cost is per run: two model calls (facts with web search, analysis without), a third with web search only when the market came back empty, plus at most one repair call per pass. The analysis pass failing never fails the run.

API ​

ProcedureWhoDoes
v1.research.statusowner, manager{ configured, provider, rendering } so the UI can say why the button is disabled
v1.research.startowner, managerTakes a ResearchSubject (kind, id, business name, optional website, area, category, notes), creates the job, enqueues it
v1.research.latestowner, managerNewest run for a subject plus the newest complete brief

Sales access to their own leads arrives with the lead tables (step 4). The subject is passed in whole because the lead and client tables do not exist yet; the job stores a frozen copy.

Changing the brief ​

The brief is versioned JSON (researchBriefSchema, v2; v1 briefs are upgraded on read with empty new sections). To add a section: bump current, add the next upgrades[n], extend the prompts, coerceFacts / coerceAnalysis and assembleBrief. Old rows keep reading. Model output is parsed tolerantly (missing lists become empty, over-long text is cut) but the stored brief always passes the strict schema.

Limits and known gaps ​

  • Instagram, TikTok and Facebook often block or login-wall rendering; their pages are then listed under "Could not read" and the model's web search fills in what it can.
  • Figures from the model's own searches are dropped (see above). A stricter source (a metrics API) would keep them; that is parked with automated post metrics.
  • A run is stopped after five minutes.
  • Re-run on a schedule, and run on lead creation or client conversion, arrive with those milestones; today a person presses "Run research".

Internal documentation. Not for clients.