Operations

Operations docs assume the deployment is already configured. Use this page to keep a running instance healthy and to diagnose component-level failures.

Health checks

Migrations

The Compose deploy profile starts a migrate service before backend, worker, UI, and codesearch. Migrate uses the owner URL and provisions ctxpipe_app. App processes use the app-role DATABASE_URL. The migrate container runs:

bun run apps/backend/src/db/migrate.ts

The AWS CDK migrate task and the hosted deploy run the same script, migrate.ts. It does these steps in this order: it makes the ctxpipe_app role ready, applies the Drizzle and OpenWorkflow migrations, runs every data backfill, and then grants ctxpipe_app access to the tables.

After the Drizzle migrations, migrate runs the backfills. The org backfill fills a missing org_id on rows that the previous release wrote (the BEFORE INSERT triggers on repository_checkouts, confluence_spaces, and claim_evidence do the same at write time). You do not run it by hand. See Upgrades for the expand-only rules and the rollback runbook.

Migrations are idempotent. Drizzle tracks applied migrations in the drizzle.__drizzle_migrations table and skips migrations that already ran. A second run changes nothing.

Migrate reads the ctxpipe_app password from DATABASE_APP_PASSWORD:

• When the password is set, migrate creates the role if it is missing and sets its password.
• When the password is not set and the role exists, migrate only grants access.
• When the password is not set and the role does not exist, migrate stops with exit code 1 before it changes the database. The error tells you how to fix it for each deployment path. Compose sends POSTGRES_PASSWORD, and the CDK migrate task sends the app database credentials secret.

For non-Compose deployments:

1. Back up Postgres first.
2. Run migrations once against the production database.
3. Start the new backend, worker, UI, and codesearch versions after migrations complete.

Logs

ctxpipe writes structured application logs through evlog. Without an OTLP logs endpoint, logs go to stdout.

Configure OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, and OTEL_EXPORTER_OTLP_METRICS_ENDPOINT on the app services when you want telemetry to leave stdout and reach your observability backend. See Configuration for headers, service names, and Langfuse. ctxpipe does not run a collector for you. Export failures are visible in your collector or vendor, not in a bundled collector container.

Backups

Back up before upgrades, secret rotation, connector migrations, and storage changes.

Chat sandbox host (AWS)

@ctxpipe/aws-cdk runs Workspace chat sandboxes on one EC2 Docker host (see AWS: Chat sandbox host). It holds no data you need to back up: a replaced host starts empty and conversations continue from their git branch.

Find the instance from the SandboxHostAutoScalingGroupName stack output and open a shell with Session Manager (no SSH key or open port is needed):

aws ssm start-session --target <instance-id>

Sandboxes stop after 5 minutes idle and are removed 30 days after last use; an organization runs at most 50 at once. A Workspace base is an image of the Workspace repository after clone and setup, so that a new conversation starts without a clone. The periodic sandbox cleanup deletes a base when a newer base replaces it, and deletes the current base when no new conversation started from it for 30 days. The same cleanup removes objects with ctxpipe labels (ai.ctxpipe.*) that no database row records. If the volume still fills, check docker system df for what grows, then raise sandboxHost.dockerVolumeSizeGiB and run cdk deploy; the new size replaces the host.

Do not run docker system prune or docker container prune while conversations are active: they delete stopped sandboxes that conversations would resume. Do not run docker image prune -a: it deletes Workspace bases, and the next new conversation of each Workspace clones the repository again while a new base builds.

Chat sandboxes (Docker Compose)

The Compose deploy profile runs Workspace chat sandboxes inside the dind service (see Docker: Chat sandboxes). Its sandbox_docker volume holds no data you need to back up. Run these from the repository root:

Sandboxes and Workspace bases are removed on the same schedule as on AWS. To free disk without touching sandboxes, remove untagged chat images and build cache:

docker compose exec dind docker image prune -f
docker compose exec dind docker builder prune -f

docker image prune without -a keeps every tagged image, so it keeps the Workspace bases. Do not run docker system prune or docker container prune inside dind while conversations are active: they delete stopped sandboxes that conversations would resume. Do not run docker image prune -a: it deletes Workspace bases, and the next new conversation of each Workspace clones the repository again while a new base builds.

Recovery runbooks

Workspace chat fails to start on Docker Compose

• A 503 saying the Docker daemon is not reachable means dind is down: confirm it is healthy (docker compose --profile deploy ps -a) and check docker compose logs dind.
• A 503 saying the chat sandbox image is missing means chat-sandbox-image has not finished or failed (for example a package mirror was unreachable, or the sandbox_docker volume was removed). Check its logs and rerun it: docker compose --profile deploy up -d chat-sandbox-image.
• Check backend logs for Docker TLS errors. After the sandbox_ca volume is removed, restart backend and worker so they read the new certificates.
• dind needs a privileged container. On a host that refuses one, chat is not available. Use the AWS CDK path, or run the sandboxes on another Docker host (see Chat sandboxes).

Workspace chat fails to start on AWS

• Confirm the sandbox host instance is InService in its Auto Scaling group.
• Check backend logs for Docker connection or TLS errors. Backend and worker reach the host at sandbox-host.ctxpipe.local:2376; the host registers that name when it finishes booting.
• If the backend reports that the chat sandbox image is missing, the host could not pull ghcr.io/ctxpipe-ai/chat-sandbox. Check the host's outbound access to ghcr.io through NAT.
• Terminate the instance to get a fresh host. The Auto Scaling group replaces it in a few minutes and reuses the stored TLS certificates.

App loads but sign-in fails

• Confirm AUTH_BASE_URL matches the browser origin.
• Confirm AUTH_SECRET is present and stable.
• Confirm provider callback URLs use the public backend origin.
• Check backend logs for Better Auth or OAuth provider errors.

Repository never becomes indexed

• Check codesearch logs for clone/index errors.
• Confirm GitHub App permissions and installation scope.
• Confirm CODESEARCH_URL points at the codesearch service from the backend.
• Reindex after fixing credentials or network access.

MCP discovery or OAuth fails

• Confirm GET /.status returns { "status": "ok" }.
• Run npx ctxpipe doctor mcp --url "$AUTH_BASE_URL/mcp".
• A ready-for-oauth result means routing, the Bearer challenge, and OAuth discovery are coherent. It does not prove authenticated tools/list or a tool call.
• Confirm AUTH_BASE_URL matches the public origin clients use, including TLS.
• Confirm the MCP URL path is /mcp.
• For authenticated OAuth, tools/list, and a list_workspaces call, use a version-pinned MCPJam client against a test organization. See MCP.

Chat or MCP returns weak context

• Confirm repositories are indexed.
• Confirm graph extraction has completed for the repository.
• Check model provider credentials and rate limits.
• Check backend and worker logs around code ingestion and graph projection.

Knowledge graph fails to load

• Confirm GRAPH_DB_URI and provider credentials.
• Confirm the graph database is reachable from backend and worker.
• Check backend logs around graph snapshot/read operations.
• Re-run repository ingestion if graph data needs to be rebuilt.

Confluence setup stalls

• Confirm Atlassian OAuth callback URL.
• Confirm Forge install URL or provisioning token.
• Confirm the worker can run Forge provisioning and Confluence sync jobs.
• See Confluence & Atlassian.

Linear setup stalls or content becomes stale

• Confirm a Linear OAuth application is registered in the wizard or that LINEAR_* is present on backend and worker.
• Confirm the Linear webhook secret is saved on the connection or LINEAR_WEBHOOK_SECRET is present on backend, and signed requests reach /api/v1/webhook/linear.
• Confirm the selected GitHub App installation still has write access and that merge pushes reach /api/v1/webhook/github/<connectionId>.
• Check worker logs for linear-sync-config, linear-sync-content, or linear-sync-incremental failures.
• Choose Sync now in the menu of the connector card after you fix the underlying credential, network, or permission failure.
• See Linear.

PagerDuty setup stalls or incidents become stale

• Confirm PagerDuty OAuth credentials are present on backend and worker.
• Confirm the connection is live and the incident's service is listed in the scope of a PagerDuty source with the status OK. There is no shared webhook secret; ctxpipe stores the per-subscription secret on the connection.
• Confirm signed requests reach /api/v1/webhook/pagerduty and merge pushes reach /api/v1/webhook/github/<connectionId>.
• Check worker logs for pagerduty-sync-config, pagerduty-sync-content, or pagerduty-sync-entity failures.
• Choose Sync now in the menu of the connector card after you fix the underlying credential, network, or permission failure.
• See PagerDuty.

Related pages