Appearance
Research (the scrape)
One pipeline researches a lead (before the call) and a client (when the account exists, and on demand). Both get the same brief, stored in research_job / research_brief, shown by the same panel. Ported from Reroot's client research and lead research briefs; the design is in ADR 0009.
flowchart LR
A[Run research] --> B[Create job + enqueue research.run]
B --> C[Render home + up to 7 pages]
C --> D[Read the code: JSON-LD, tel, mailto, platform, colours]
D --> E[Render up to 5 social profile pages]
E --> F[Pass 1: facts, model with web search]
F --> G{Market empty?}
G -->|yes| H[Pass 1b: focused market search]
G -->|no| I
H --> I[Pass 2: analysis, no tools]
I --> J[Assemble: page code beats the model]
J --> K[Verify figures, store brief v2]
What it finds
Brief v2 has twelve sections. The workspace page shows each, says how many have data and names the ones with nothing.
| Section | Source |
|---|---|
| Overview: summary, tagline, founded, team, languages, brand colours | Site text and code (theme-color, JSON-LD), then web search |
| Contact and hours: phones, emails, addresses, hours, booking link | Page code first (tel/mailto links, schema.org JSON-LD), then web search |
| What they sell: offerings, products with prices, USPs, how customers buy | Site pages (menu, services, shop, pricing) |
| Reputation: rating, review count, sentiment, quotes, awards, press | JSON-LD aggregateRating, then web search (Google, Trustpilot, press) |
| Website: platform, booking / ordering / video, pages, problems | Page code (platform signature, forms, video embeds) and the analyst |
| Social accounts: handle, bio, followers, posting cadence, themes | Links on the site, profile pages, web search |
| Videos | Video and reel links; web search |
| Content and ideas: themes, gaps, video ideas with hooks | Analyst pass |
| Audience: personas | Analyst pass |
| Market: competitors, similar businesses, creators, trends, seasonality, price position | Web search; a focused extra pass if pass 1 found none |
| Sales prep: recommended package, quick wins, objections, talking points | Analyst pass, choosing only from the agency's real packages |
| Evidence: pages read and failed, profiles read, passes, structured data | The run itself |
Facts read from page code override the model: a phone number in a tel: link beats one the model remembers. Socials and videos found on the site are never dropped by the model.
Never invent figures. A follower or view count is kept only if that number appears in the text of a page this run rendered. Counts the model saw only in its own searches cannot be checked, so they are removed (the link stays). Every URL must be a well-formed public http(s) address; private and loopback hosts are refused, both for rendering and in the output.
Where it runs
| Piece | Where | Notes |
|---|---|---|
| Page rendering | Cloudflare Browser Rendering, the BROWSER binding (Cloudflare.Browser("BROWSER") in alchemy.run.ts) | quickAction("content") returns the HTML after scripts ran, so JS-heavy sites are read like a visitor sees them. Images, media, fonts and styles are skipped. |
| The run | The jobs queue, job type research.run v1 (limit 8 minutes) | Not in a request: it takes about a minute. Idempotent: a finished job is left alone, so a retry changes nothing. |
| Model | Injected: see below | |
| Storage | D1 | History is kept; the newest complete brief is "current" and stays visible during a re-run. |
The model backend (dependency injection)
The pipeline depends on three small interfaces in packages/api/src/research/ports.ts: PageRenderer, LlmProvider, ResearchStore. The server is the injection root (apps/server/src/services.ts: getResearch); tests pass fakes.
| Backend | Use | How |
|---|---|---|
claude-cli | Development only: the Claude app, signed in as you | A Worker cannot spawn a process, so bun run claude:bridge runs a small loopback server that calls your claude CLI with WebSearch and WebFetch only. Same idea as Reroot's ClaudeCliProvider: your own login, no API key. |
anthropic | Production: the Claude Platform | Messages API with an API key and the server-side web_search tool. |
Selection (select.ts): RESEARCH_LLM = auto (default), claude-cli, anthropic or off. auto prefers the API key, then the bridge, then nothing. Production never uses the bridge: on the production stage it is refused even if configured. With nothing configured, starting research is refused with the exact reason; there is no fake brief.
Development
bash
bun run claude:bridge # prints CLAUDE_BRIDGE_URL and CLAUDE_BRIDGE_TOKEN
# put both lines in apps/server/.env, then:
bun run devThe bridge binds to 127.0.0.1, requires the token, never gives the CLI Bash/Write/Edit/Read, and runs at most two calls at once. Set CLAUDE_BRIDGE_MODEL to change the model (default sonnet).
Production
Set ANTHROPIC_API_KEY (and optionally ANTHROPIC_MODEL, default claude-sonnet-4-5) as deployment secrets. A secret is only bound to the Worker when it is set at deploy time, so a deploy without a key still works and research reports "not configured". Cost is per run: two model calls (facts with web search, analysis without), a third with web search only when the market came back empty, plus at most one repair call per pass. The analysis pass failing never fails the run.
API
| Procedure | Who | Does |
|---|---|---|
v1.research.status | owner, manager | { configured, provider, rendering } so the UI can say why the button is disabled |
v1.research.start | owner, manager | Takes a ResearchSubject (kind, id, business name, optional website, area, category, notes), creates the job, enqueues it |
v1.research.latest | owner, manager | Newest run for a subject plus the newest complete brief |
Sales access to their own leads arrives with the lead tables (step 4). The subject is passed in whole because the lead and client tables do not exist yet; the job stores a frozen copy.
Changing the brief
The brief is versioned JSON (researchBriefSchema, v2; v1 briefs are upgraded on read with empty new sections). To add a section: bump current, add the next upgrades[n], extend the prompts, coerceFacts / coerceAnalysis and assembleBrief. Old rows keep reading. Model output is parsed tolerantly (missing lists become empty, over-long text is cut) but the stored brief always passes the strict schema.
Limits and known gaps
- Instagram, TikTok and Facebook often block or login-wall rendering; their pages are then listed under "Could not read" and the model's web search fills in what it can.
- Figures from the model's own searches are dropped (see above). A stricter source (a metrics API) would keep them; that is parked with automated post metrics.
- A run is stopped after five minutes.
- Re-run on a schedule, and run on lead creation or client conversion, arrive with those milestones; today a person presses "Run research".