Appearance
Operations
Day-to-day checks, what each alert means, and what to do. Deploy steps are in Deployment.
Daily checks
- Open the owner dashboard. Alerts appear as notifications linking to
/owner. GET /healthreturnsstatus: "ok".degradedmeans email is not configured or has failed rows.GET /readyreturnsready: true.
Alerts
The alerts.ops sweep runs every 15 minutes. Each alert is raised at most once per 24 hours, so a standing problem does not flood the bell. A sweep that has never run does not alert.
| Alert | Meaning | Action |
|---|---|---|
| Email failed to send | One or more emails reached the failed state | Open /health for the count, check the Cloudflare Email dashboard and CLOUDFLARE_EMAIL_API_KEY, then retry |
| Email is stuck in the queue | Queued emails are overdue for delivery | Check the cron trigger for the email flush (every minute) and Cloudflare Email status |
| Payment amount mismatch | A payment came back with a different amount than the invoice | Read the payment.mismatch audit row, compare with Stripe, and do not mark the invoice paid by hand |
| Background sweeps have stopped | No sweep run was recorded for over an hour | Check the 15-minute cron trigger and the Worker logs for [jobs] errors |
| Email is not configured | Production runs without CLOUDFLARE_EMAIL_API_TOKEN or CLOUDFLARE_ACCOUNT_ID | Set both and redeploy |
| Payments are not configured | Production runs without STRIPE_SECRET_KEY | Set the secret and redeploy |
The two configuration alerts only appear in production.
Logs
Workers Logs is enabled on the server Worker. Filter by event:
request:requestId,role(anonymousbefore sign-in),method,route,status,ms,slow(over 2 seconds).error: the samerequestId, the errorname, and a message cut to 300 characters.
Quote the requestId when someone reports a 500 error. The response carries the same id as requestId and x-request-id.
Public link abuse
Public links (intake, call booking, quote and contract pages) are limited per caller address and per token. A caller who goes over the route budget gets RATE_LIMITED with a "try again in a few minutes" message. Budgets are in CALLER_POLICY in packages/api/src/public-shared.ts. Repeated refusals in the logs for one address are the signal to watch; the address comes from cf-connecting-ip.
Backups and restore
- Run
D1_DATABASE=<name> bun run ops:backupat least weekly, and copy the file off the machine. - Run the restore drill in
scripts/ops/restore-d1.mdmonthly.
Smoke test
bun run ops:smoke https://<server origin> after every deploy. It exits non-zero on any failure.
Rotating secrets
Rotate CLOUDFLARE_EMAIL_API_TOKEN or STRIPE_SECRET_KEY in the provider dashboard, then set the new value and redeploy. Rotating BETTER_AUTH_SECRET signs every owner and crew member out. Follow Rotating OWNER_PASSWORD and BETTER_AUTH_SECRET.
Incident steps
- Note the time, the
requestIdvalues, and the alert titles. - Check
/health,/ready, and the smoke test. - Read the matching
errorlog lines, then the audit rows for the affected records. - If data is wrong, stop and restore from backup per the runbook before further writes.
- Write a short record of what happened and what changed.