Skip to content

Operations ​

Day-to-day checks, what each alert means, and what to do. Deploy steps are in Deployment.

Daily checks ​

  1. Open the owner dashboard. Alerts appear as notifications linking to /owner.
  2. GET /health returns status: "ok". degraded means email is not configured or has failed rows.
  3. GET /ready returns ready: true.

Alerts ​

The alerts.ops sweep runs every 15 minutes. Each alert is raised at most once per 24 hours, so a standing problem does not flood the bell. A sweep that has never run does not alert.

AlertMeaningAction
Email failed to sendOne or more emails reached the failed stateOpen /health for the count, check the Cloudflare Email dashboard and CLOUDFLARE_EMAIL_API_KEY, then retry
Email is stuck in the queueQueued emails are overdue for deliveryCheck the cron trigger for the email flush (every minute) and Cloudflare Email status
Payment amount mismatchA payment came back with a different amount than the invoiceRead the payment.mismatch audit row, compare with Stripe, and do not mark the invoice paid by hand
Background sweeps have stoppedNo sweep run was recorded for over an hourCheck the 15-minute cron trigger and the Worker logs for [jobs] errors
Email is not configuredProduction runs without CLOUDFLARE_EMAIL_API_TOKEN or CLOUDFLARE_ACCOUNT_IDSet both and redeploy
Payments are not configuredProduction runs without STRIPE_SECRET_KEYSet the secret and redeploy

The two configuration alerts only appear in production.

Logs ​

Workers Logs is enabled on the server Worker. Filter by event:

  • request: requestId, role (anonymous before sign-in), method, route, status, ms, slow (over 2 seconds).
  • error: the same requestId, the error name, and a message cut to 300 characters.

Quote the requestId when someone reports a 500 error. The response carries the same id as requestId and x-request-id.

Public links (intake, call booking, quote and contract pages) are limited per caller address and per token. A caller who goes over the route budget gets RATE_LIMITED with a "try again in a few minutes" message. Budgets are in CALLER_POLICY in packages/api/src/public-shared.ts. Repeated refusals in the logs for one address are the signal to watch; the address comes from cf-connecting-ip.

Backups and restore ​

  • Run D1_DATABASE=<name> bun run ops:backup at least weekly, and copy the file off the machine.
  • Run the restore drill in scripts/ops/restore-d1.md monthly.

Smoke test ​

bun run ops:smoke https://<server origin> after every deploy. It exits non-zero on any failure.

Rotating secrets ​

Rotate CLOUDFLARE_EMAIL_API_TOKEN or STRIPE_SECRET_KEY in the provider dashboard, then set the new value and redeploy. Rotating BETTER_AUTH_SECRET signs every owner and crew member out. Follow Rotating OWNER_PASSWORD and BETTER_AUTH_SECRET.

Incident steps ​

  1. Note the time, the requestId values, and the alert titles.
  2. Check /health, /ready, and the smoke test.
  3. Read the matching error log lines, then the audit rows for the affected records.
  4. If data is wrong, stop and restore from backup per the runbook before further writes.
  5. Write a short record of what happened and what changed.

Internal documentation. Not for clients.