# Self-host ctxpipe (https://docs.ctxpipe.ai/docs/self-hosting) Self-hosted ctxpipe is the same product as hosted ctxpipe. It runs in infrastructure that you control. The ctxpipe team is not in the data path. Your deployment holds the credentials and the source data. Model requests go from your deployment to the provider that you choose. ## Choose a deployment path | Path | Use it when | You operate | | ------------------------------------------------------ | --------------------------------------------------------------------- | --------------------------------------------------- | | [Docker Compose](/docs/self-hosting/deployment/docker) | You evaluate ctxpipe, or run a small internal deployment on one host. | One Compose project and its volumes | | [AWS CDK](/docs/self-hosting/deployment/aws) | You run ctxpipe in your own AWS account. | The `CtxPipe` construct, a domain, and model access | Each path provides the same services in a different way: | | Docker Compose | AWS CDK | | --------------- | -------------------------------------- | ------------------------------------------- | | Postgres | `pgvector/pgvector:pg17` container | Aurora PostgreSQL 16 | | Graph database | FalkorDB container | Amazon Neptune | | Knowledge store | ClickHouse container | ClickHouse on one EC2 host, old parts on S3 | | Sandboxes | Docker-in-Docker (`dind`) | A Graviton EC2 Docker host | | Public origin | Your TLS proxy in front of the backend | Route 53 and a load balancer | Start with the [quickstart](/docs/self-hosting/quickstart). It covers each path. ## What you run * **Backend**: auth, the REST API, the MCP endpoint, webhooks, and the public origin. It also serves the UI. * **UI**: the web app, behind the backend. * **Worker**: the background jobs for ingestion, connectors, and Workspace writes. * **Codesearch**: repository clones and code search indexes, on persistent storage. * **PostgreSQL with pgvector**: the application data and the job state. * **A graph database**: FalkorDB by default. * **ClickHouse**: the new knowledge store for the documents, their sections, and their relations. See [Knowledge store](/docs/self-hosting/configuration#knowledge-store). * **A sandbox host**: runs Workspace chat and ingestion in isolated containers. Without one, chat answers `503`. For another platform, see [Chat sandboxes](/docs/self-hosting/configuration#chat-sandboxes). OpenTelemetry export is off by default. See [Observability](/docs/self-hosting/configuration#observability). ## What you own * One stable public HTTPS origin, `AUTH_BASE_URL`, for the browser, the API, auth, MCP, and the provider callbacks. * Durable storage, backups, and recovery for Postgres, the graph database, ClickHouse, and codesearch. See [Graph databases](/docs/self-hosting/graph-databases). * The secrets for auth and encryption, and the model provider credentials for chat, ingestion, and embeddings. See [Models](/docs/self-hosting/models). * A provider app for each connector that you turn on: a GitHub App for repositories, and OAuth apps for Atlassian, Slack, Linear, Notion, and PagerDuty. See the source connector pages in this section. * Monitoring, migrations, upgrades, and rollback. After the deployment is healthy, people use it like hosted ctxpipe. Send them to [Connections](/docs/connections) and [Connect your agent](/docs/mcp/mcp-docs). # Connections (https://docs.ctxpipe.ai/docs/connections) A **connection** gives ctxpipe access to one of your tools. A **source** tells one [Workspace](/docs/workspaces) what to read through a connection. ctxpipe writes the content into the Workspace repository as Markdown files, so you can read, review, and keep each file that ctxpipe and your agents use. * A **connection** holds only the authorization: the OAuth grant, API access token, or GitHub App installation, and its webhooks. An organization owner or admin adds it one time for the organization, on **Connectors**. * A **source** is one Markdown file in a Workspace, at `sources//.md`. It names the connection, the scope (for example Linear teams or Notion pages), and an optional description. One connection can feed many Workspaces. Each Workspace has its own source file and its own scope. ## Available connectors | Connector | Type | What enters the Workspace | Scope | | ------------------------------------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------- | | [GitHub](/docs/connections/source-connectors/github) | Git source | Linked repositories, indexed for code search and extraction, and their issues and closed pull requests | One repository, and an optional `ref` | | [Linear](/docs/connections/source-connectors/linear) | Scoped mirror | Issues, projects, cycles, and related content | Teams and projects | | [Notion](/docs/connections/source-connectors/notion) | Scoped mirror | Pages and databases | Pages and databases | | [Confluence](/docs/connections/source-connectors/confluence) | Scoped mirror | Spaces and pages | Spaces, with all pages or a page list | | [PagerDuty](/docs/connections/source-connectors/pagerduty) | Scoped mirror | Incidents of the selected services | Services | | [Buildkite](/docs/connections/source-connectors/buildkite) | Scoped mirror | The latest 25 finished builds of the selected pipelines | Pipelines | | [Plain](/docs/connections/source-connectors/plain) | Scoped mirror | Help Center articles and the newest threads of selected labels | Help Centers and thread labels | | [Slack](/docs/connections/source-connectors/slack) | Capture on request | One thread at a time, when someone mentions the bot | Channels | | [Plain git](/docs/workspaces/linked-repositories#plain-git-repositories) | Git source | A public repository on any git host, with no connection | A URL | ctxpipe indexes the docs in linked repositories with the code: README files, `docs/**`, ADRs, `AGENTS.md`, and skills. The main difference is the scope. A scoped mirror keeps an approved subset current. Slack has no history backfill and no channel mirror: a person captures one thread when the team needs it later. ## Connect GitHub first The ctxpipe GitHub App lets ctxpipe read and write Workspace repositories, index linked code repositories, and open the pull requests for source changes, upgrades, and Workspace chat. The other connectors write into a Workspace repository on GitHub, so they need it too. See the [GitHub connector](/docs/connections/source-connectors/github). ## Add a tool to a Workspace Before you start, make sure that: * you are an organization owner or admin; * the ctxpipe GitHub App can write to the Workspace repository, and the repository uses format 2 or higher (see [Upgrades](/docs/workspaces/settings#upgrades)); * you can merge pull requests in the Workspace repository. ### Connect the tool Open **Connectors**, choose **Add connection**, then choose **Connect** on the row of the tool. To find a tool, type its name in the search field or select its group. A tool that is not available yet has **Ask for it**. Follow the authorization steps for the tool. Buildkite uses an API access token. The connector page has the details for each tool. ### Add it to Workspaces The next step is **Add to Workspaces**. Select the Workspaces, then select the scope. For more than one Workspace, choose **Same scope for each Workspace** or **A different scope for each Workspace**. Then choose **Add to Workspaces**. You can do this later. Expand the connector on **Connectors** and choose **Add to a Workspace** (or **Manage**), or use **Add** under **Sources** in the **Settings** of a Workspace. ### Merge the pull requests ctxpipe opens one pull request for each Workspace. Each pull request adds the source file. Review the scope and who can read the repository, then merge. The source does not sync before the merge. ### Wait for the first sync After the merge, ctxpipe makes the first mirror. The Workspace then indexes the new files like any other knowledge file. To change what a Workspace reads, edit, rename, or remove its source. See [Workspace settings](/docs/workspaces/settings#sources). ## How content stays current * **Webhooks.** Linear, Notion, Confluence, PagerDuty, and Plain tell ctxpipe when content changes. ctxpipe updates the files of the sources whose scope includes the change. Events that arrive before a source is live are skipped. * **Buildkite webhooks.** A finished build starts a full sync after a 10-minute window. More events in that window share the same sync. * **Daily full sync.** Once a day, ctxpipe syncs each connection that has a source with the status **Connected**. This sync finds content that the provider deleted, and content that a webhook missed. * **Sync now.** The menu of the connector on **Connectors** has **Sync now** for Confluence, Linear, Notion, PagerDuty, Buildkite, and Plain. It starts the same full sync at once. Only an owner or admin sees it. Do not edit mirrored files by hand. The next sync overwrites them. When you remove a source, the next sync deletes its files in one commit. Git history keeps them. **Remove connector**, in the same menu, removes the authorization. The sources of the connection stop syncing. The source files and the files that ctxpipe committed stay in git. ## Git is the review boundary A source file changes only through a pull request that a person merges. The pull request is the review of the scope. No agent can add or change a source without that review. * **Scoped mirrors** write Markdown under the folder of the provider, for example `linear/` or `notion/`, in each Workspace repository. * **Slack** captures commit directly to the default branch of each Workspace that has the channel. The mention in the thread is the approval, and git history records it. Each person who can read the Workspace repository can read its mirrored content. Use separate Workspaces when different sources need different audiences. ## Scope limits Each source has a maximum number of items, so that one sync stays inside the API rate limits of the provider. For more items, add a second source with the same connection. | Source | Maximum in one source | | -------------- | --------------------------------------------------------------------------- | | **Confluence** | 100 spaces, and 500 selected pages in one space (select all pages for more) | | **Slack** | 100 channels | | **Linear** | 50 teams and 100 projects | | **Notion** | 100 root pages and 100 databases | | **PagerDuty** | 100 services | | **Buildkite** | 50 pipelines | | **Plain** | 50 Help Centers and 50 thread labels | A GitHub source holds one repository, so it has no limit. ## Troubleshooting | Problem | What to do | | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Authorization fails | Allow popups and try again. On a self-hosted deployment, ask the operator to check the OAuth app of the provider and its callback URL. | | **Add to Workspaces** shows an error | The message names the cause: **Needs the format upgrade first.** next to a Workspace (see [Upgrades](/docs/workspaces/settings#upgrades)), **This Workspace is read-only** (see [Write access](/docs/workspaces/settings#write-access)), or **Ask an admin to choose what this source syncs** (your role). | | No pull request appears | Make sure that the GitHub App can write to the Workspace repository. Close the dialog and try again. | | The source status is not **Connected** | Read the status and the reason in **Settings**. See [Source status](/docs/workspaces/settings#source-status). | | Changes in the tool do not arrive | Choose **Sync now**. If changes still do not arrive, ask the operator to check the webhook of the provider. | | A sync fails | Choose **Sync now**. If it fails again, ask the operator to check the worker logs, the GitHub write access, and outbound access to the API of the provider. | | The authorization was revoked | Reconnect the tool from its connector card. Updates stop until you do. | ## More connectors We plan more connectors for engineering systems, for example Jira and Sentry. On a self-hosted deployment, the operator must configure the OAuth app and the webhooks of a provider before its connector works. See the source connector pages under [Self-hosting](/docs/self-hosting). Buildkite uses a token for each connection. See [Self-host Buildkite](/docs/self-hosting/buildkite). ## Agent access is separate MCP clients are not connections. Claude Code, Cursor, CI agents, and other MCP clients connect to ctxpipe through the MCP endpoint. See [Connect your agent](/docs/mcp/mcp-docs). # Common workflows (https://docs.ctxpipe.ai/docs/getting-started/common-workflows) Each workflow starts from the same Workspace. People use [Workspace chat](/docs/workspaces/chat). Coding agents use the [MCP tools](/docs/mcp/use-from-your-agent). ## Plan a change across repositories Link the repositories that a team owns to its Workspace. Before an agent plans a change, it searches the knowledge for the decisions and standards that apply, then searches the code in each linked repository. The [knowledge graph](/docs/workspaces/graph) shows which services, APIs, and libraries the change touches. ```text Use ctxpipe before you plan this change. Find the decisions and standards that apply, and the code in other repositories that calls this API. ``` ## Give every agent the same instructions Keep `AGENTS.md`, skills, ADRs, and contribution guides in linked repositories or in the Workspace repository. ctxpipe returns them with the code that they govern. Claude Code, Cursor, Codex, and your review bots then follow the same guidance as your engineers. One MCP endpoint serves each client, so you do not build retrieval for each agent. ## Bring product and incident context into git Linear, Notion, Confluence, PagerDuty, Buildkite, and Plain mirror the scope that you approve into the Workspace. Slack captures a thread when someone mentions the bot. The result is Markdown with a git history, not a hidden index. An agent can cite the issue or the incident behind a decision. See [Connections](/docs/connections). ## Write it down once When chat or an agent finds a fact that the team needs later, ask Workspace chat to write it as a knowledge file. Review the diff and open a pull request. The next person and the next agent start from that file. [Local memory](/docs/mcp/local-memory) does the same for the lessons that agents learn in one repository. ## Run headless agents and review bots CI jobs, review bots, and other agents that cannot sign in through a browser use an organization API key. See [API keys](/docs/mcp/api-keys). ## Run ctxpipe on your own infrastructure The open-source core runs on Docker Compose, Terraform, or AWS CDK. Your source data and model calls stay in your infrastructure. See [Self-hosting](/docs/self-hosting). # Get support (https://docs.ctxpipe.ai/docs/getting-started/get-in-touch) ## Product and deployment questions * [Book a 30-minute call](https://cal.com/ctxpipe/30min) for product, deployment, or commercial questions. * [Request hosted early access](https://ctxpipe.ai/early-access). * Visit [ctxpipe.ai](https://ctxpipe.ai) for the product overview. ## Community and technical support * Join the [ctxpipe Discord](https://ctxpipe.ai/discord) for questions and discussion. * Report reproducible bugs or request features through [GitHub issues](https://github.com/ctxpipe-ai/ctxpipe/issues). * Follow [@ctxpipe on X](https://x.com/ctxpipe) for product updates. ## Documentation and source * [Documentation](https://docs.ctxpipe.ai) * [Source code](https://github.com/ctxpipe-ai/ctxpipe) * [Open-source overview](/docs/resources/open-source) # Getting started with ctxpipe (https://docs.ctxpipe.ai/docs/getting-started) ctxpipe is context management for software factories. A software factory is the set of coding agents, and the harnesses that run them, that a team uses to write software. Your coding agents can reach every repository, ticket, doc, and Slack thread. They cannot tell which parts are true. ctxpipe gives them one organized, current, and inspectable source of context: a git repository of Markdown that you own, and that you can read, review, and correct. 1. **Connect.** [Link your code repositories](/docs/workspaces/linked-repositories), and [select what to bring in](/docs/connections) from Linear, Notion, Confluence, PagerDuty, Buildkite, Plain, and Slack. ctxpipe writes it as Markdown into one git repository for each [Workspace](/docs/workspaces). 2. **Keep current.** Background jobs mirror changes from your tools, and read a linked repository again after each push. Each job makes [one small commit](/docs/workspaces#what-ctxpipe-writes) that you can review or revert. Then ctxpipe [indexes the new commit](/docs/workspaces/indexing) for search and the graph. 3. **Serve.** Claude Code, Cursor, Codex, OpenCode, and VS Code read the same Workspace [through one MCP endpoint](/docs/mcp/mcp-docs). ctxpipe returns sources, not answers. Your agent reasons with its own model. ## Who ctxpipe is for ctxpipe is for engineering teams that run coding agents across many teams and repositories, and for the platform engineers who build the agent harness. It helps when: * agents repeat mistakes that a decision record or a runbook already explains; * the context for one change is in several repositories and tools; * you must see what an agent read, and correct it in one place. For the full problem, the teams that ctxpipe is not for, and how ctxpipe compares with other approaches, see [ctxpipe.ai](https://www.ctxpipe.ai). ## Key terms | Term | Meaning | | ------------------------ | ----------------------------------------------------------------------------------- | | **Organization** | Your team in ctxpipe. It holds members, connections, and Workspaces. | | **Workspace** | One body of knowledge, for example one product or one team. | | **Workspace repository** | The git repository of Markdown that holds the knowledge of a Workspace. You own it. | | **Linked repository** | A code repository that a Workspace indexes for search. | | **Connection** | The authorization for one tool, such as Linear or Notion. | | **Source** | A file in the Workspace repository that says what to read through a connection. | ## Choose your setup ## Give these docs to your agent Point your coding agent at [docs.ctxpipe.ai/llms.txt](https://docs.ctxpipe.ai/llms.txt). It lists each page as Markdown. For all the pages in one file, use [llms-full.txt](https://docs.ctxpipe.ai/llms-full.txt). To give the agent one page, choose **Copy Markdown** under the title of that page. ## Learn more # Hosted quickstart (https://docs.ctxpipe.ai/docs/getting-started/quickstart) This guide sets up hosted ctxpipe with one Workspace, and connects it to your coding agent. The agent connection is the goal. Workspace chat is a quick check that the Workspace has the right knowledge. We help you set up your first Workspace. [Request early access](https://ctxpipe.ai/early-access) or [book a demo](https://cal.com/ctxpipe/30min). To run ctxpipe on your own infrastructure, start with the [self-hosting quickstart](/docs/self-hosting/quickstart). ## Before you start You need: * permission to install the ctxpipe GitHub App on your GitHub account or organization, or a GitHub owner who can approve it; * a GitHub repository for the knowledge of the Workspace. An empty repository is good. You can create one during setup; * the code repositories to link; * an MCP client: Claude Code, Cursor, Codex, OpenCode, or VS Code with Copilot. ## 1. Sign up and create your organization Sign up at [app.ctxpipe.ai](https://app.ctxpipe.ai), then create an organization. With a work email, ctxpipe suggests a name from your email domain. If a teammate invited you, you join their organization. **Home** then shows the **Get started** guide. It lists the next steps and marks each step done when ctxpipe sees the result. ## 2. Connect GitHub In the guide, choose **Connect GitHub**. Install the ctxpipe GitHub App on the account or organization that owns your repositories. Give it access to the repository for the knowledge, and to the code repositories that you want to link. You can add repositories later in GitHub. A GitHub account can belong to one ctxpipe user only. See [GitHub connector](/docs/connections/source-connectors/github). ## 3. Create a Workspace Choose **Create Workspace**, then choose the Workspace repository: * **Select GitHub**: use a repository that you have on GitHub. * **Create on GitHub**: create a new repository on GitHub, then select it. * **Paste URL**: use any other git URL. ctxpipe cannot write to it, so the Workspace is read-only. ctxpipe adds `AGENTS.md` and a knowledge skill if they are missing, and starts to read the repository. See [Create a Workspace](/docs/workspaces/create-workspace). ## 4. Link code repositories The next step shows the **Sources** of the new Workspace. Choose **Add repositories**, select the code repositories, and choose **Open a pull request**. Merge the pull request on GitHub. Each repository then goes from **Pending** to **Indexing** to **Indexed**. To link more repositories later, open the **Settings** tab of the Workspace. See [Linked repositories](/docs/workspaces/linked-repositories). ## 5. Ask a question Type a question in the Workspace, and ask for the files behind the answer: ```text What should a new engineer read first, and which files support that answer? ``` You can ask while ctxpipe indexes. The answers improve when indexing is done. See [Workspace chat](/docs/workspaces/chat). ## 6. Connect your coding agent The **Get started** guide has a **Use ctxpipe from your agents** block with the setup command. Copy it, and run it in a repository: ```bash npx ctxpipe init ``` The CLI signs you in, asks which organization and agent clients to configure, and writes the MCP configuration. The first time your agent connects, it asks you to authorize ctxpipe in the browser. See [Connect your agent](/docs/mcp/mcp-docs). Then give the agent a task: ```text Use ctxpipe before you plan this change. Search the knowledge for the standards and decisions that apply, search the code for the related areas, and say what still needs a check in the code. ``` ## 7. Connect your tools and invite your team * Open **Connectors** to add Linear, Notion, Confluence, PagerDuty, Buildkite, Plain, or Slack. See [Connections](/docs/connections). * Open **Organization settings → Members** to invite your team. See [Organizations and members](/docs/managing-organization). ## Setup is complete when * the linked repositories show **Indexed** in the **Settings** of the Workspace; * Workspace chat answers a question and names useful files; * your agent can call `list_workspaces` and `search_knowledge`. # Organizations and members (https://docs.ctxpipe.ai/docs/managing-organization) An **organization** is the access boundary for Workspaces, connections, conversations, and MCP. A person can belong to more than one organization, and switch between them with the organization switcher. * Each organization has a name and a unique URL slug. * Workspace and connector pages always work inside the organization in the URL. * Each member of an organization can open each of its [Workspaces](/docs/workspaces). ## Invite by email Open **Organization settings → Members** and send an invitation. The invited person gets an email with a link. * If they have an account, the link adds them to the organization with the role of the invitation. * If they do not, they sign up first with the same email. ctxpipe then applies the invitation. ## Roles Each member has one role in each organization: | Role | What the role can do | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------- | | **owner** | All that an admin can do. Also delete the organization. | | **admin** | Manage members and invitations. Add and remove connections, and start a sync. Start Workspace upgrades. Create organization API keys. | | **member** | Open and use the Workspaces of the organization, and chat in them. See the status of connections and upgrades. | Admin actions need a signed-in browser session. An API key or an MCP token cannot do them, even when its owner is an admin. ## Access follows membership and source scope Membership gives a person access to the organization in ctxpipe. It does not give access to more content in your tools: * the GitHub App installation controls which code ctxpipe can read; * the scope of each source controls which Linear, Notion, Confluence, PagerDuty, Buildkite, or Plain content ctxpipe mirrors; * the permissions of the Workspace repository control who can read the mirrored content in git; * Slack captures need the bot in the channel. Use roles to control administration in ctxpipe. Keep the permissions of your tools, and the audience of each Workspace repository, as narrow as you need. An MCP client reads only the organizations that you selected for it, and only while you are a member. See [Connected apps](/docs/mcp/connected-apps). # API keys (https://docs.ctxpipe.ai/docs/mcp/api-keys) OAuth is the default for a person at a laptop. Use an API key only when a client cannot finish the browser sign-in: a CI job, a headless agent, a review bot, or a host that accepts only a fixed header. ## Choose a key | | Organization key | Personal key | | ------------------ | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | Where to create it | **Organization settings → API Keys** | **User account → API Keys** | | Who can create it | An organization owner or admin | Each user, for themselves | | Acts as | The organization, not the person who created it | You, as a member | | Reads | Its one organization. The `org` argument must name it. | Each organization that you are a member of. ctxpipe checks your membership on each call. | | Works on | `/mcp` only | `/mcp`, and the REST API as you | Use an organization key for shared automation. It continues to work when the person who created it leaves the organization. A new key expires after 30 days by default, and allows 1,000 requests per hour. You can create a key that never expires, but treat it as a long-lived secret. ctxpipe shows the secret one time only. Give each key a name. To rotate a key, create a new key, change `CTXPIPE_API_KEY` to it, then revoke the old key. ## Send the key Send the key in the `x-api-key` header to `/mcp`. If a host can send only a Bearer token, send `Authorization: Bearer `. A Bearer API key works only on `/mcp`, not on the REST API. Keep the secret out of files. Put a reference to the `CTXPIPE_API_KEY` environment variable in the client configuration. The reference is safe to commit. The CLI writes that reference for you: ```bash npx ctxpipe mcp add --client cursor --scope both --auth api-key --non-interactive ``` The CLI never creates a key and never reads `CTXPIPE_API_KEY`. Start from the client configuration in [Configure a client by hand](/docs/mcp/mcp-docs#configure-a-client-by-hand), and add the header entry for your client: | Client | File | Header entry | | ----------------- | ----------------------------------------------------- | --------------------------------------------------------------------------- | | Claude Code | `.mcp.json` | `"headers": { "x-api-key": "${CTXPIPE_API_KEY}" }` | | Cursor | `.cursor/mcp.json` or `~/.cursor/mcp.json` | `"headers": { "x-api-key": "${env:CTXPIPE_API_KEY}" }` | | Codex | `.codex/config.toml` or `~/.codex/config.toml` | `env_http_headers = { "x-api-key" = "CTXPIPE_API_KEY" }` | | OpenCode | `opencode.json` or `~/.config/opencode/opencode.json` | `"headers": { "x-api-key": "{env:CTXPIPE_API_KEY}" }`, and `"oauth": false` | | VS Code / Copilot | `.vscode/mcp.json` | `"headers": { "x-api-key": "${env:CTXPIPE_API_KEY}" }` | Each client uses its own syntax for the variable. Codex needs the name of the variable, not `$VAR` and not the secret. In OpenCode, `"oauth": false` stops the client from opening the browser after a `401`. For example, a Claude Code `.mcp.json` with an API key: ```json { "mcpServers": { "ctxpipe": { "type": "http", "url": "https://app.ctxpipe.ai/mcp", "headers": { "x-api-key": "${CTXPIPE_API_KEY}" } } } } ``` ## Set `CTXPIPE_API_KEY` The process of the MCP client must have the variable. An `export` in the terminal where you ran the CLI is not enough for an app that you start from the Dock or the Start menu. * **CLI agents** (Claude Code, Codex, OpenCode, Cursor CLI): export the variable in the terminal, then start the agent from that terminal. To set it for each session, put it in `~/.zshenv` or `~/.bash_profile`. The Cursor CLI can start a shell that does not read `.zshrc`. * **Desktop apps** (Cursor, VS Code): set the variable for your user session, then quit and open the app again. On macOS, run `launchctl setenv CTXPIPE_API_KEY ''`. On Linux, use `environment.d`. On Windows, use the user environment variables. You can also start the app from a terminal that has the variable, for example `cursor .`. * **CI**: store the key as a secret of your CI system, and expose it to the job as `CTXPIPE_API_KEY`. To test the key, run: ```bash CTXPIPE_API_KEY= npx ctxpipe doctor mcp --url "https://app.ctxpipe.ai/mcp" ``` The status `ready-for-api-key` means that the key works. ## Troubleshooting * **An organization key gets `401` on the REST API.** This is correct. Organization keys work only on `/mcp`. Use a personal key or a browser session for the REST API. * **A tool gets `org_not_found`.** An organization key reads only its own organization. Make the `org` argument match it. * **The client sends no key.** A variable in the environment is not enough by itself. The client configuration must map it to the `x-api-key` header. # Connected apps (https://docs.ctxpipe.ai/docs/mcp/connected-apps) When an MCP client signs in to ctxpipe with OAuth, you approve it on a consent page. **Connected apps** lists each app that you approved. Use it to change what an app can read. You do not need to reconnect. ## Open the page Open your account settings and choose **Connected apps**. On hosted ctxpipe, the page is at `https://app.ctxpipe.ai/.auth/account/connected-apps`. A self-hosted deployment uses the same path on its own origin. For each app, the page shows the client name, what it can do, and the organizations that it can read. **Read** is the normal access for an MCP client. **Changes** shows only when you also ticked **Also allow changes** on the consent page. ## Choose the organizations On the consent page, ctxpipe lists your organizations with check boxes. The organization of your current session is selected. Select at least one. An app can read only the organizations in its set. When you join a new organization, ctxpipe does not add it to the apps that you approved before. To change the set: 1. Choose **Edit organizations** for the app. 2. Select or clear organizations. 3. Save. The change applies on the next tool call. The app does not need to reconnect. ## Revoke an app Choose **Revoke** for the app. The app loses access at once, and it must ask you to authorize it again. An older connection that holds one organization shows **Reconnect to choose more organizations**. You can only revoke it. ## When a tool says `org_not_found` The tool call named an organization that the connection cannot read. One of these is true: * the organization is not in the set of the app. Add it with **Edit organizations**, then call the tool again; * you are no longer a member of the organization; * the slug is wrong. Call `list_workspaces` with no `org` to see the organizations of the connection. An organization API key reads only its own organization. Any other `org` gives the same error. ## API keys API keys are not apps, so they are not on this page. See [API keys](/docs/mcp/api-keys). # Local memory (https://docs.ctxpipe.ai/docs/mcp/local-memory) Local memory builds on features of coding agents (Cursor, Claude Code, Codex, OpenCode, VS Code) that change from month to month, and on ideas about agent memory that are still settling. Expect this page and `ctxpipe memory init` to change frequently. After you upgrade the CLI, run `npx ctxpipe memory init` again to get the latest hooks, rules, and skills. ## What it is Local memory is a folder of Markdown files, `.ai/memory/`, that lives in your repository. Coding agents read it when they start a task and add to it when they learn something worth keeping: a convention, a decision, or a correction from an engineer. It is plain files in Git. There is no database and no background service, and nothing leaves your machine until you commit and push. ## Why it exists Agents forget. Every new chat starts from scratch. Engineers then repeat the same corrections ("we use pnpm here", "don't call that API directly"). Local memory keeps those lessons: * **Across chats and tools.** What one agent learns, the next one reads, whether it runs in Cursor, Claude Code, Codex, OpenCode, or VS Code. * **With your team.** Memory is committed and reviewed like code, so a lesson one engineer teaches reaches everyone once it merges. * **In ctxpipe.** When the repository is linked to a Workspace, merged lessons become part of the knowledge of that Workspace. The MCP read tools also pass them on to agents that work in other repositories. ## How it works 1. **The agent notices.** When you tell an agent something that sounds lasting ("from now on…", "we decided…"), a small hook saves it as a *candidate* in `.ai/memory/events/`. Candidates stay on your machine. That folder is gitignored. 2. **The agent decides.** When the agent finishes its turn, the hook shows it the candidates. The agent writes the ones worth keeping into the right file (for example `lessons-learned.md`, or a new decision record) and discards the rest. Hooks never write lasting memory themselves. 3. **You commit it with the work.** Memory changes go in the same commit and pull request as the code they came from. If memory stays uncommitted, the agent gets one reminder. It gets another reminder if a later commit leaves it out. 4. **The pull request tells the story.** The agent summarizes the work in the pull request description: what changed, why, what was tried and ruled out, and what is left. That description is reviewed with the code. 5. **ctxpipe reads what merges.** After the merge, ctxpipe reads `lessons-learned.md` as instructions for how your team works, and decision records as decisions. Where pull request mirroring is on, it also reads pull request descriptions as the history behind each change. ## What each agent gets Paths are for repository scope. With `--scope user`, the same files go in your home directory. | Agent | Capture and reminders | Instructions | | ----------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------- | | Cursor | Hooks in `.cursor/hooks.json` | `.cursor/rules/ai-memory.mdc` and skills in `.cursor/skills/` | | Claude Code | Hooks in `.claude/settings.json` | `.claude/rules/ai-memory.md` and skills in `.claude/skills/` | | Codex | Hooks in `.codex/config.toml` | A block in `AGENTS.md` | | VS Code (Copilot) | Hooks in `.github/hooks/ctxpipe-memory.json` | `.github/copilot-instructions.md` and `.github/instructions/ctxpipe-memory.instructions.md` | | OpenCode | Plugin in `.opencode/plugins/ctxpipe-memory.js` | `.opencode/memory-capture.md`, `AGENTS.md`, and an `instructions` entry in `opencode.json` | `memory init` always writes the Cursor rule and skills in the repository, for each choice of agents. * **Codex** runs project hooks only after you trust the project and approve the hooks in `/hooks`. * **VS Code** agent hooks are in preview. * **OpenCode** has no end-of-turn hook yet, so the plugin posts the reminder as a new message after the agent finishes. * **Cursor and VS Code** can also run Claude Code hooks from `.claude/settings.json`. ctxpipe detects this so capture runs once per turn. ## Install ```bash npx ctxpipe memory init --agents cursor,claude --scope repo --non-interactive ``` Or with the main setup wizard, which also configures the ctxpipe MCP: ```bash npx ctxpipe init --org acme --agents cursor,claude --scope repo --memory --non-interactive ``` `--agents` takes any of `cursor`, `claude`, `codex`, `opencode`, and `vscode`. `--scope` is `repo`, `user`, or `both`. Without prompts, the default is `repo`. The interactive wizard asks. ## What's in the folder ``` .ai/memory/ README.md # how agents should use this folder index.md # map of the stores below lessons-learned.md # confirmed conventions and corrections glossary.md # project terms product-context.md # product and architecture overview decisions/ # decision records (ADR-*.md) + index.md PRDs/ # product requirements + index.md sessions/ # summaries of work that has no pull request events/ # candidate inbox, gitignored ``` ## Local memory and the ctxpipe MCP | | What it does | | -------------------------- | --------------------------------------------------------------------------------------- | | `ctxpipe` MCP (read tools) | Searches and reads your Workspaces: their knowledge files, connectors, and linked code. | | Local memory (this page) | Markdown in one repository plus agent hooks. It is not an MCP server. | You can use local memory on its own (`memory init`) or alongside the ctxpipe MCP (`init --memory`). Local memory keeps the lessons of one repository. When you link that repository to a Workspace, the MCP serves the merged lessons to every agent in your organization. ## Finding things in memory Agents start from `index.md` and each store's own `index.md`. For anything else they search with ripgrep, skipping the candidate inbox: ```bash rg -i "keyword" .ai/memory --glob '*.md' --glob '!events/**' ``` The installed `memory-search` skill describes the full procedure. ## Checking and upgrading * `npx ctxpipe memory status` and `npx ctxpipe memory doctor` show whether memory is set up, and list the memory files that are not committed yet. * The hooks call `npx ctxpipe memory capture`. Agents use `memory capture promote ` and `memory capture dismiss ` to keep or discard a candidate. You do not run these commands yourself. * Re-run `npx ctxpipe memory init` after upgrading the CLI. It refreshes the rules, skills, and hooks, and keeps your own memory files. It also removes retired pieces from older versions: the `ctxpipe-memory` MCP entry, old memory skills, and an AgentMemory-era `README.md`. It also moves `patterns.md` into `lessons-learned.md`. # Connect your agent (https://docs.ctxpipe.ai/docs/mcp/mcp-docs) ctxpipe has one remote MCP server for every organization: ```text https://app.ctxpipe.ai/mcp ``` A self-hosted deployment uses the same `/mcp` path on its own origin. The server uses Streamable HTTP. It gives your agent [seven read-only tools](/docs/mcp/use-from-your-agent). ## Install with the CLI Run this in a repository: ```bash npx ctxpipe init ``` The wizard signs you in, asks which organization and agent clients to configure, and shows the planned file changes before it writes them. Write the configuration to the repository, to your user profile, or to both. To share the setup with your team, commit the repository configuration that the CLI writes. It holds no secret. The wizard can also set up [local memory](/docs/mcp/local-memory). To configure the MCP only, run `npx ctxpipe mcp add`. | Client | Repository configuration | User configuration | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Claude Code** | `.mcp.json` | Runs `claude mcp add` when the Claude CLI is installed. Without the Claude CLI, the CLI writes the repository `.mcp.json`, or prints the command for an API key. | | **Cursor** | `.cursor/mcp.json` | `~/.cursor/mcp.json` | | **Codex** | No repository file. Prints a `codex mcp add` command, which writes your user configuration. For an API key, it prints the `.codex/config.toml` lines. | Runs `codex mcp add` when the Codex CLI is installed. Without the Codex CLI, the CLI prints the command. For an API key, it prints the `config.toml` lines. | | **OpenCode** | `opencode.json` | `~/.config/opencode/opencode.json` | | **VS Code / Copilot** | `.vscode/mcp.json` | Prints a VS Code install link | The CLI sign-in is only for setup. The CLI keeps it in the OS keychain, or in `~/.config/ctxpipe/` when no keychain is available. Your agent client asks you to authorize ctxpipe separately the first time it connects. ### Scripts and CI Use flags to run the CLI without prompts: ```bash npx ctxpipe init --org acme --agents cursor,claude --scope repo --non-interactive npx ctxpipe mcp add --client opencode --scope user --non-interactive npx ctxpipe doctor --json ``` Run `npx ctxpipe init --help` for all flags. Examples are `--base-url`, `--dry-run`, `--json`, `--no-mcp`, and `--auth api-key`. For an agent that cannot sign in through a browser, see [API keys](/docs/mcp/api-keys). ## Authorize the client The first time your client connects, it opens ctxpipe in the browser: 1. Sign in to ctxpipe. 2. Select the organizations that the client can read. The organization of your current session is selected. Select at least one. 3. Leave **Also allow changes** clear. It shows only when a client asks for write access, and the MCP tools never need it. 4. Approve. The client returns to your agent, ready to use. Each tool call names one of the organizations that you selected. To change the set later, or to revoke the client, open [Connected apps](/docs/mcp/connected-apps). ## Configure a client by hand Use manual configuration when the CLI does not support your client, or when another tool manages the configuration. The client opens the browser to authorize, so the configuration holds no token. ### Claude Code `.mcp.json` in the repository: ```json { "mcpServers": { "ctxpipe": { "type": "http", "url": "https://app.ctxpipe.ai/mcp" } } } ``` For your user profile, run `claude mcp add --transport http ctxpipe --scope user https://app.ctxpipe.ai/mcp`. ### Cursor `.cursor/mcp.json` in the repository, or `~/.cursor/mcp.json`: ```json { "mcpServers": { "ctxpipe": { "type": "http", "url": "https://app.ctxpipe.ai/mcp" } } } ``` ### Codex ```bash codex mcp add ctxpipe --url "https://app.ctxpipe.ai/mcp" ``` ### OpenCode `opencode.json` in the repository, or `~/.config/opencode/opencode.json`: ```json { "mcp": { "ctxpipe": { "type": "remote", "url": "https://app.ctxpipe.ai/mcp", "enabled": true } } } ``` ### VS Code / Copilot `.vscode/mcp.json` in the repository: ```json { "servers": { "ctxpipe": { "type": "http", "url": "https://app.ctxpipe.ai/mcp" } } } ``` For your user profile, use the install link that `npx ctxpipe mcp add --client vscode --scope user` prints. ### Other clients Use the endpoint URL with the Streamable HTTP transport. The name of the transport key changes between clients. Read the documentation of your client. ## Check the connection Ask your agent: ```text Call the ctxpipe list_workspaces tool, then search_knowledge for our standards on authentication. ``` A finished browser sign-in does not prove tool access. The tools must return your Workspaces. ## Troubleshooting * **The client asks for authorization after the CLI signed you in.** This is correct. The CLI sign-in and the client authorization are separate. * **The connection fails with only a URL.** Add the `type` field for your client. Most clients do not detect the transport. * **Cursor or Claude Code refuses the server type.** Use `"type": "http"`. * **OpenCode shows the server, but it does not connect.** Use the top-level `mcp.ctxpipe` shape with `type: "remote"` and `enabled: true`. * **The client cannot finish the browser sign-in.** Use an [API key](/docs/mcp/api-keys). * **A tool returns `org_not_found`.** See [Connected apps](/docs/mcp/connected-apps#when-a-tool-says-org_not_found). Old configurations can have `?orgSlug=` in the URL. The server ignores it, so you can keep it or remove it. The current CLI writes the URL without it. ## Related * [Use ctxpipe from your agent](/docs/mcp/use-from-your-agent): the tools and how to use them * [API keys](/docs/mcp/api-keys): CI, review bots, and headless agents * [Connected apps](/docs/mcp/connected-apps): change organizations or revoke a client * [Local memory](/docs/mcp/local-memory): Markdown memory that agents keep in the repository # Use ctxpipe from your agent (https://docs.ctxpipe.ai/docs/mcp/use-from-your-agent) The ctxpipe MCP gives a coding agent seven read-only tools. The agent does the reasoning and drives the search. ctxpipe returns facts with stable refs, and runs no model for a tool call. To connect a client, see [Connect your agent](/docs/mcp/mcp-docs). ## The tools | Tool | What it does | Main arguments | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | `list_workspaces` | Lists the organizations and Workspaces that the connection can read. With one Workspace, it also gives its repositories, sources, and readiness. | `org`, `workspace` | | `search_knowledge` | Searches the knowledge with natural language or keywords. | `query`, `workspaces`, `kinds`, `pathPrefix`, `sources`, `updatedAfter`, `validAt` | | `search_code` | Searches the linked repositories in one call. | `query`, `mode` (`literal`, `regex`, or `structural`), `repositories`, `language`, `pathGlob`, `caseSensitive` | | `find_symbol` | Finds the definition, references, callers, or callees of a symbol. | `symbol`, `relation`, `filePath`, `language` | | `read_file` | Reads a knowledge file or a repository file by its ref. | `ref`, `startLine`, `endLine` | | `list_files` | Lists the folders and files of a Workspace repository or a linked repository. | `repository`, `path`, `depth`, `glob` | | `get_related` | Finds the knowledge near one or more seeds in the graph that is relevant to a question. The results are ranked and explained. | `seeds`, `query`, `intent`, `excludePredicates`, `excludeKinds` | Each tool has the read-only hint. The tools never change your data. The `structural` mode of `search_code` needs `language`. For `get_related`, `intent` is one of `why`, `ownership`, `change`, `dependencies`, `structure`, `provenance`, or `general`. The server also gives each file as a resource, `ctxpipe://{org}/{workspace}/{ref}`. A client that supports resources lets you @-mention a file in your prompt. ## Organizations and Workspaces * Each call names one organization in the `org` argument. Use the slug from your ctxpipe URL. Only `list_workspaces` can leave it out, and then it lists each organization of the connection. * The tools that read one Workspace take its `workspace` slug. `search_knowledge` takes a `workspaces` list. Without a list, it searches all Workspaces of the organization that are ready, up to 8. If the organization has more than 8, name the Workspaces. * When a session starts, the server sends instructions that list your organizations and their Workspaces. In most cases, the agent can call a tool at once. * An organization outside the connection gives `org_not_found`. See [Connected apps](/docs/mcp/connected-apps#when-a-tool-says-org_not_found). ## Refs A ref is a short, stable name for a result. Use a ref from one tool in another. * A knowledge file ref is its path in the Workspace repository, for example `knowledge/services/payments-api.md`. * A repository file ref is `/:`, for example `example/payments-api:src/limits.ts`. Each result shows the branch and the commit that ctxpipe served. ctxpipe serves the default branch of each repository, or the pinned branch or tag. ## Exclude the repository that you work in Your agent works on a local branch with newer files than ctxpipe has. The agent must read that repository from disk, not from ctxpipe. `search_code` and `find_symbol` therefore need `excludeRepositories`: put the URL or the handle of your current repository in it, or send `[]` to exclude nothing. `read_file` and `list_files` take the same list, as an option. A ref into an excluded repository gives `excluded_repository`. ## Good habits for an agent * Search the knowledge before a design choice. Ask for standards, decisions, and owners. * Send calls that do not depend on each other in parallel. * Use `search_code`, then `read_file` on the refs that it returns. * Use `get_related` with a ref that you found, and a `query` or an `intent`, to see why a decision was made or who owns a service. * Cite the refs in your answer, and say what still needs a check in the code. ```text Use ctxpipe before you plan this change. Call list_workspaces, then search_knowledge for the standards on authentication, and search_code for the session handling code. Exclude the repository that we work in. ``` ## Limits and errors * Each response has a size limit. When a result does not fit, the tool returns one page and a `cursor`. Send the cursor to get the next page. A cursor stops working when the Workspace changes, and the tool returns `stale_cursor`. Then search again. * An error has `isError: true` and `{ code, message, retryable }`. * Each tool needs the `ctxpipe:read` scope. API keys and the normal OAuth grant have it. A call without it gets `403 insufficient_scope`. The MCP tools never need `ctxpipe:write`. ## Related * [Connect your agent](/docs/mcp/mcp-docs): install and authorize a client * [Connected apps](/docs/mcp/connected-apps): change organizations or revoke a client * [Workspaces](/docs/workspaces): where the knowledge is # Data and privacy (https://docs.ctxpipe.ai/docs/resources/data-processing) This page is about **hosted ctxpipe**. On a self-hosted deployment, you choose the providers and the retention. The model calls go from your deployment to the provider that you configure. The ctxpipe team never sees your prompts or your model traffic. ## What ctxpipe reads * The git repositories that you link to a Workspace, and the Workspace repository itself. * The content of the sources that you add. This is: * Linear, Notion, Confluence, PagerDuty, Buildkite, and Plain mirrors. * The issues and closed pull requests of linked GitHub repositories. * The Slack threads that a person captures. ctxpipe writes this content into the Workspace repository, which you control. Then it indexes the content. ctxpipe reads nothing that you did not connect. ## What you control * You create each Workspace, link each repository, and approve each source scope in a pull request. ctxpipe ingests nothing without one of these actions. * ctxpipe writes only to Workspace repositories. It never writes to your code repositories. See [What ctxpipe writes](/docs/workspaces#what-ctxpipe-writes). * You can unlink a repository, remove a source, or delete a Workspace at any time. This stops new indexing. The indexed data can stay until the deletion process of your organization is complete. [Get in touch](/docs/getting-started/get-in-touch) for the details. * The data of your organization is separate from the data of other organizations. * If you stop using ctxpipe, your Workspace repositories stay with you, with their full history. ## Sub-processors | Provider | Purpose | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Amazon Web Services (AWS), us-west-2 | Compute and hosting, the Postgres database, storage, logs, and traces. Traces, which may hold prompt text, are kept 30 days. Isolated sandboxes (microVMs) for Workspace chat and for ingestion. A sandbox holds a copy of the Workspace repository, or of the linked repository that it ingests. | | Cloudflare | The edge network: DNS, TLS, the proxy that each request to ctxpipe passes through, the tunnels to AWS, and Access for our internal tools | | OpenRouter | Model routing, only to zero data retention endpoints | | Anthropic | AI assistants used by the ctxpipe team for engineering operations; may process production logs and traces, outside OpenRouter, so zero data retention does not apply | | OpenAI | AI assistants used by the ctxpipe team for engineering operations; may process production logs and traces, outside OpenRouter, so zero data retention does not apply | | Railway | Being retired. Compute and hosting, and our own observability (logs, traces, and model traces), until 2026-11-15 | | Neon | Being retired. Postgres database until 2026-11-15 | | Vercel | Being retired for sandboxes. Isolated sandboxes for Workspace chat and for ingestion until 2026-11-15 | ## How ctxpipe uses models * **Workspace chat.** Your messages, and the context that the agent retrieves (search results, file excerpts, and graph context), go to the model. See [Where the agent runs](/docs/workspaces/chat#where-the-agent-runs). * **Ingestion.** ctxpipe uses models to extract knowledge from parts of the linked repositories, in a sandbox with a closed network. See [Where extraction runs](/docs/workspaces/indexing#where-extraction-runs). * **MCP tools.** The tools call no model. They search and read your data. Your coding agent uses its own model. ctxpipe does not send a whole repository to a model in one request. Retrieval and tools select the context. On hosted ctxpipe, each model call goes through OpenRouter to [zero data retention (ZDR)](https://openrouter.ai/docs/guides/features/zdr) endpoints only. Our OpenRouter account enforces this, so the model providers do not store your prompts or train on them. See the [OpenRouter privacy policy](https://openrouter.ai/privacy). ## Retention We do not publish fixed retention periods for indexed data yet. One exception: ctxpipe deletes the sandbox files of a Workspace chat conversation when its sandbox stops, 5 minutes after the last use of the conversation. The branch of the conversation in your repository stays. For deletion times, export, or a DPA, [get in touch](/docs/getting-started/get-in-touch). ## What we do not do * We do not train models on your content. * We do not share data across organizations, and we do not sell it. The sub-processors above only operate the product. * We do not ingest sources that you did not connect. ## Certifications SOC 2 and ISO 27001 for ctxpipe are on the roadmap. Our infrastructure sub-processors publish their own compliance and security documents. For questions about your data, sub-processors, or contracts, [get in touch](/docs/getting-started/get-in-touch). # FAQ (https://docs.ctxpipe.ai/docs/resources/faq) ## Product ### What is ctxpipe? Context management for software factories: the coding agents, and the harnesses that run them, that a team uses to write software. ctxpipe keeps your code, decisions, tickets, docs, and incidents as Markdown in a git repository that you own. It keeps the repository current with background jobs. It serves the repository to each coding agent through one MCP endpoint. See [Getting started](/docs/getting-started). ### How is ctxpipe different from giving an agent more MCP servers? More servers give an agent more to read, not a way to tell what is true. ctxpipe gives each agent the same reviewed context: files in git with sources, history, and links to the code. Your agent cites the files that it used. See [Use ctxpipe from your agent](/docs/mcp/use-from-your-agent). ### Does ctxpipe train on my code? No. On hosted ctxpipe, each model call goes only to zero data retention endpoints. See [Data and privacy](/docs/resources/data-processing). ## Connections ### Which sources can I connect? GitHub, Linear, Notion, Confluence, PagerDuty, Buildkite, Plain, and Slack, plus public repositories on any git host. Linear, Notion, Confluence, PagerDuty, Buildkite, and Plain mirror the scope that you approve. Slack captures one thread when someone mentions the bot. See [Connections](/docs/connections). ### Is ctxpipe only for GitHub? ctxpipe writes Workspace repositories through the GitHub App, and reads private code through it. Other git hosts work as read-only sources: ctxpipe reads a public repository on any host with no credential. A private repository on another host does not work yet. See [Plain git repositories](/docs/workspaces/linked-repositories#plain-git-repositories). ### What is the difference between a connection and a source? A connection is the authorization for one tool. A source is a file in a Workspace that names the connection and the scope. One connection can feed many Workspaces. ### Does ctxpipe scan my laptop? No. ctxpipe reads only the repositories and tools that you connect. It installs nothing on developer machines. [Local memory](/docs/mcp/local-memory) is Markdown in your repository, and it leaves your machine only when you push it. ### Does ctxpipe write to my code repositories? No. ctxpipe only reads linked code repositories, and writes only to Workspace repositories. See [Code repositories stay read-only](/docs/workspaces#code-repositories-stay-read-only). ## MCP ### How does a tool call choose the organization? The `/mcp` URL is the same for each organization. Each tool call names one organization in the `org` argument. ctxpipe allows the call only when the organization is in the set that you selected for the client, and you are still a member. The tools have no organization id that an agent can change. See [Use ctxpipe from your agent](/docs/mcp/use-from-your-agent). ### Which transport does the MCP use? Streamable HTTP. A client that supports only the old SSE transport can fail. ### Can an agent change data through the MCP? No. All seven tools are read-only. ## Account and organization ### How do invitations work? An owner or admin sends an email invitation from **Organization settings → Members**. See [Invite by email](/docs/managing-organization#invite-by-email). ### Can one person belong to many organizations? Yes. Switch with the organization switcher. The URL always names the organization. An MCP client can read each organization that you select for it. ### Can I verify my company domain? Not yet. Access to an organization comes from invitations and roles. If you need domain verification, [get in touch](/docs/getting-started/get-in-touch). ## Self-hosting ### What is the fastest way to evaluate a self-hosted deployment? Docker Compose: `pnpm start` runs the `deploy` profile. See [Docker](/docs/self-hosting/deployment/docker). ### Which databases does ctxpipe need? PostgreSQL with pgvector, and a graph database. FalkorDB is the default. Neo4j, Memgraph, and Amazon Neptune also work. See [Graph databases](/docs/self-hosting/graph-databases). ### Which model providers work? Each OpenAI-compatible provider, for example OpenRouter, Azure OpenAI, or Ollama. Amazon Bedrock also works. See [Models](/docs/self-hosting/models). # Open source (https://docs.ctxpipe.ai/docs/resources/open-source) ctxpipe is open source under the **MIT License**. The core includes each job that writes to your Workspace, so you can read exactly what changes your knowledge. You can use, copy, change, and distribute the software. The `LICENSE` file in the repository is the binding text. This page is not legal advice. ## Repository The source is at [github.com/ctxpipe-ai/ctxpipe](https://github.com/ctxpipe-ai/ctxpipe), a pnpm and Turborepo monorepo: | Path | What it is | | -------------------------- | ----------------------------------------------------------------------------------------- | | `apps/backend` | The API, the MCP server, auth, the database schema, and the background jobs (Hono on Bun) | | `apps/ui` | The web app (TanStack Start and React) | | `apps/codesearch` | Code search, code navigation, and repository checkout (Zoekt and SCIP) | | `apps/forge-ctxpipe-agent` | The Atlassian Forge app for Confluence | | `apps/docs` | This documentation site | | `packages/cli` | The `ctxpipe` CLI for MCP and local memory setup | | `packages/aws-cdk` | The AWS CDK construct for self-hosting | | `plugins/ctxpipe` | The Claude plugin | | `examples/` | Runnable examples, such as an AWS CDK deployment | ## Hosted and self-hosted Hosted ctxpipe runs the same open-source core. Self-host it to control your data and infrastructure. Or use hosted ctxpipe, where we operate it for you. See [Self-hosting](/docs/self-hosting). ## Contribute You can open issues and pull requests on GitHub. Read `CONTRIBUTING.md` for local development. Before a large change, read the architecture decisions in `.ai/memory/decisions/`, so that your proposal agrees with them. ## Report a security problem Do not report a security problem in a public issue, and do not post secrets. Use a private GitHub security advisory on the repository, or the contact channels on [ctxpipe.ai](https://ctxpipe.ai). # Workspace chat (https://docs.ctxpipe.ai/docs/workspaces/chat) Workspace chat is a coding agent that works on one Workspace. It answers from the knowledge files, the knowledge graph, and the linked code of the Workspace, and it can edit the Workspace repository for you. ## Start a conversation Open a Workspace and type in the composer. Your first message creates the conversation and gives it its own URL. The sidebar shows your five most recent conversations under each Workspace. ```text I am new here. What are the main services, and which files should I read first? ``` ```text Which services depend on the billing events stream, and where is that implemented? ``` ```text Before we change authentication, which decisions and standards apply? ``` To stop a reply, choose **Stop**. If a reply stops before it ends, choose **Retry** to send your question again. ## How chat finds answers Each answer can use: * the **knowledge files** of the Workspace, and the content that connectors mirror; * the **knowledge graph** built from the links and claims in those files; * **code search** across the [linked repositories](/docs/workspaces/linked-repositories); * the earlier messages of the conversation. Chat sees only this Workspace. It does not search other Workspaces of your organization. ## Change files Ask the agent to write or update knowledge: ```text Write a knowledge unit for the checkout service from what you found, and link it to the payments service. ``` While the agent works: * **Files** shows the Workspace repository as this conversation sees it. You can edit, create, rename, and delete files there yourself. * **Diff** shows all the changes of the conversation, or the changes of one turn. Use **Split view** to compare the two versions side by side. Each turn that changes files also shows a card with its changes in the thread. Each conversation works on its own branch, `ctxpipe/chat//`, and never pushes to the default branch. The agent commits its work when it finishes a task. ## Publish with a pull request The header of the conversation has these actions: * **Sync** (the cloud-upload icon) shows when the sandbox has changes that GitHub does not have. It commits all changes in the sandbox, yours and the agent's, and pushes the branch of the conversation. * **Create PR** opens a pull request from the branch of the conversation to the default branch. * **Show PR** opens that pull request while it is open. Later pushes go into the same pull request. Review and merge the pull request like any other. After the merge, the Workspace indexes the new commit, and the next message continues on a new branch (`…/`) from the new default branch. A pull request that is closed without a merge keeps its branch. Choose **Create PR** again to open a new one. If the default branch moves while a conversation is open, ctxpipe brings the conversation up to date before the next message. Until then, a conversation with its own changes shows **The Workspace has new commits**, and you cannot publish it. Send a message to bring it up to date. If the update has a conflict, the agent resolves it. ## Where the agent runs Each conversation runs in its own isolated sandbox, with a copy of the Workspace repository. The first message can take longer while the sandbox starts (**Setting up sandbox**). Later messages use the same sandbox. On hosted ctxpipe, each conversation gets its own [Amazon Bedrock AgentCore](https://aws.amazon.com/bedrock/agentcore/) session, in its own microVM. No other conversation or organization shares it. * **Commits stay.** A sandbox stops after 5 minutes without use. Before it stops, ctxpipe pushes the commits that are not on GitHub to the branch of the conversation. Your next message starts a new sandbox from that branch. Changes that nobody committed are deleted. * **Closed network.** The agent can reach GitHub, the ctxpipe model proxy, and the Workspace tools. It cannot reach another host. * **No credential is in the sandbox.** A proxy outside the sandbox adds each credential to a request on its way out. The GitHub token can only read. It is short-lived, and ctxpipe revokes it when the sandbox stops. Access to the model and the Workspace tools is added only while a reply runs. The sandbox never holds a credential that can push to your default branch. Model calls, Workspace tools, and pushes go through ctxpipe, limited to that conversation. ## Sandbox limits An organization runs at most 50 sandboxes at once. Chats, background jobs, and the preparation of Workspace sandboxes all count. Ingestion can use at most 40, so chat always has 10. When your organization is at the limit, chat shows **Workspace chat is at capacity**. Wait a few minutes and try again. An ingestion that waits shows **Waiting for a free sandbox**, and starts by itself when a slot is free. These limits keep the work of an organization fair. They are not spend limits. If they are not enough for your team, [contact us](/docs/getting-started/get-in-touch). ## Check important answers Chat can miss a file, read an index that is a few minutes behind, or state a relationship too strongly. For important answers: * ask which files support the answer; * ask what is verified and what is inferred; * check the indexing status in **Settings**; * read the current code before a risky change. Each person who can read the Workspace repository can read the content that connectors mirrored into it. The access settings of the repository therefore also control who can read captured Slack threads and mirrored Linear, Notion, or Confluence content. ## Use the same context from your coding agent Coding agents read the same Workspaces through the [ctxpipe MCP](/docs/mcp/mcp-docs). The MCP tools search and read the Workspace directly. They do not start a Workspace chat, so no MCP call adds a conversation to the sidebar. # Create a Workspace (https://docs.ctxpipe.ai/docs/workspaces/create-workspace) Add Workspace has two steps. Step 1 chooses the **Workspace repository**. When you choose it, the Workspace exists and ctxpipe starts to read it. Step 2 adds the code repositories and tools that the Workspace reads. Step 2 is optional. ## Before you start * Connect GitHub on **Connectors** if you want to select or create a repository on GitHub. ctxpipe can write only to the repositories that the ctxpipe GitHub App can access. * Choose the repository for the knowledge of this Workspace. An empty repository is good. An existing documentation repository is good too. ## Create the Workspace ### Open Add Workspace Choose **+** next to **Workspaces** in the sidebar. You can also choose **Create Workspace** on Home, or on the GitHub setup page after you connect GitHub. ### Choose the repository Choose one of these options: * **Select GitHub**: search the repositories that the ctxpipe GitHub App can see. * **Create on GitHub**: open the new-repository page of GitHub. Create the repository, make sure that the App can access it, then return and select it. * **Paste URL**: use any git URL. ctxpipe cannot write to a repository that is not on GitHub, so that Workspace is read-only. To keep the Workspace in one folder of a larger repository, type the folder in **Root folder**, for example `docs/context`. See [Root folder](#root-folder). Choose **Create Workspace**. ### Add code repositories and tools ctxpipe checks that it can write to the repository and sets it up. Then it shows the **Sources** of the new Workspace: * Choose **Add repositories** under **GitHub** to link [code repositories](/docs/workspaces/linked-repositories). * Choose **Add** next to a connected tool, such as Linear or Notion. * Choose **Add another connection** (or **Add connection**) to connect a new tool. This closes Add Workspace. Each change opens a pull request that adds a source file under `sources/`. The change applies after you merge it. You can make the same changes later in **Settings**. Choose **Open Workspace** (or **Done**) to finish. If the repository has ctxpipe content from an earlier version, the step shows **Upgrade this Workspace before you add sources**. See [Upgrades](/docs/workspaces/settings#upgrades). If ctxpipe cannot write, the step shows the reason. See [Write access](/docs/workspaces/settings#write-access). ### Wait for the first index ctxpipe reads the default branch and builds the index. You can start a conversation at once. The answers improve when indexing is done. A repository can be the Workspace repository of only one Workspace in an organization. Other Workspaces can link it as a linked repository. ## What ctxpipe adds to the repository On a writable repository, ctxpipe makes one setup commit that adds these files if they are missing: * `AGENTS.md`, with the Workspace `name` in its front matter and a short folder map; * `.agents/skills/ctxpipe-knowledge/SKILL.md`, which tells coding agents how to write knowledge files for this Workspace; * `.ctxpipe/workspace.yaml`, which holds the [repository format](/docs/workspaces/settings#repository-format), on a repository that has no ctxpipe content yet. ctxpipe keeps the files that exist. It adds only what is missing. ## Root folder The root folder is the folder of the repository that holds the Workspace. The default is the root of the repository. * ctxpipe reads and writes only below the root folder. This applies to the knowledge files, `AGENTS.md`, `sources/`, `.ctxpipe/workspace.yaml`, connector mirrors, ingestion, and conversation commits. * The paths in **Files**, in conversations, and in the MCP tools are relative to the root folder. * ctxpipe refuses a conversation push that changes a file outside the root folder. * The root folder is a relative path, such as `docs/context`. It cannot start with `/` or contain `..` or `.git`. It can contain only letters, numbers, `.`, `_`, `-`, and `/`. When you change the root folder in **Settings**, ctxpipe reads the Workspace again from the new folder. The current index serves until the new one is ready. Files in the old folder stay where they are. # Knowledge graph (https://docs.ctxpipe.ai/docs/workspaces/graph) The **Graph** tab shows the knowledge graph of the current Workspace. Each node is a knowledge file, and each edge comes from a link or a claim in those files. See [Knowledge files](/docs/workspaces/knowledge-files). The knowledge that ctxpipe extracts from linked code is in knowledge files too, so it is in the graph. The graph shows after the first index of the Workspace. While ctxpipe builds it, the tab shows **Graph, building**. ctxpipe builds the graph from files. To fix the graph, fix the files: correct a claim, add a link, or set `kind`. Then push. The graph updates when the Workspace re-indexes. ## Read the graph * Nodes are colored by `kind` and sized by how connected they are. * Edges are links (`LINKS_TO`) and claims (`DEPENDS_ON`, `PART_OF`, `OWNS`, and so on). * The activity timeline groups relationships by when they became valid. Drag across it to filter the graph to a time range. Use the canvas controls to zoom, reset the view, pause or play the layout, lasso-select a group of nodes, or capture a screenshot. ## Inspect a node Select a node to open its details: * **Path**: the file behind the node. **Go to definition** opens it in the **Files** tab; **Copy path** copies it. * **Summary**, **Connections**, and **Strongest connections**. * **Predicates** and **Neighbor kinds**: what kinds of relationships and nodes surround it. * **Activity**: when its relationships were valid. Review a node with few or no connections. It is often a stub or a stale unit. Lasso-select several nodes to see a summary of the selection: node and edge counts, kinds, and predicates. ## Treat the graph as a guide The graph shows what the files claim. Confidence and dates help you decide where to look, but they are not proof. Before you act on an important relationship, open the file and the source it points to. # Workspaces (https://docs.ctxpipe.ai/docs/workspaces) A **Workspace** holds one body of team knowledge, for example one product area or one team. Each Workspace has: * one **Workspace repository**: a git repository of Markdown knowledge files, connector content, and `AGENTS.md`; * zero or more **linked repositories**: code repositories that ctxpipe indexes, so that chat and agents can search them. An organization can have many Workspaces. Each member of the organization can open each Workspace. ## Git is the source of truth All that ctxpipe knows about a Workspace is in files in its Workspace repository. ctxpipe reads those files to build its search index and knowledge graph, and writes new knowledge back as ordinary commits. Because of this: * you read, review, and edit the knowledge with the tools that you already use; * each change that ctxpipe makes is a commit in the history of the repository; * if you stop using ctxpipe, the repository and its knowledge stay with you. ### What ctxpipe writes ctxpipe writes only to the Workspace repository, through the ctxpipe GitHub App. Each change is a small commit to the default branch, or a pull request: | Change | How it gets to the default branch | | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- | | The [setup commit](/docs/workspaces/create-workspace#what-ctxpipe-adds-to-the-repository) of a new Workspace | A commit | | Connector mirrors: Linear, Notion, Confluence, Slack, PagerDuty, Buildkite, Plain, and the issues and closed pull requests of linked GitHub repositories | A commit for each sync | | [Knowledge extracted](/docs/workspaces/indexing#how-extraction-works) from linked repositories | One commit for each run, `Update extracted knowledge` | | Upkeep, for example links that ctxpipe rewrites after a rename, a `valid_from` that a file does not have, or the automatic-upgrades setting | A commit | | A change to `sources/`: add, edit, rename, or remove a source | A pull request that a person merges | | A [format upgrade](/docs/workspaces/settings#upgrades) | A pull request on `ctxpipe/upgrade/format-` | | [Workspace chat](/docs/workspaces/chat#publish-with-a-pull-request) | A branch for each conversation, and a pull request that you open | ctxpipe never force-pushes the default branch. It force-pushes only refs that it owns: the branch of a conversation, the upgrade branch, and the temporary refs of an extraction run. If someone else pushed to the default branch first, ctxpipe rebases its change onto theirs. If both changed the same lines, ctxpipe merges the two changes before it pushes. If the default branch is protected, connector mirrors and extracted knowledge stop. See [Write access](/docs/workspaces/settings#write-access). ### Code repositories stay read-only ctxpipe does not write to [linked repositories](/docs/workspaces/linked-repositories). It reads them with short-lived tokens that can only read. The GitHub mirror only reads issues and pull requests, and writes them to the Workspace repository. A linked repository can also be the Workspace repository of another Workspace in your organization. ctxpipe then writes to it only for that other Workspace, and does not extract knowledge from it. ## Use a Workspace Open a Workspace from the sidebar. The main column is the conversation. The pane on the right has these tabs: | Tab | What it shows | | ------------ | ----------------------------------------------------------------------------------------------------------------------- | | **Files** | The Workspace repository, as the current conversation sees it. Browse, edit, create, rename, and delete files. | | **Diff** | The changes of the current conversation. It shows only when a conversation is open. | | **Graph** | The knowledge graph of the Workspace. While ctxpipe builds it, the tab shows **Graph, building**. | | **Settings** | The name, repositories, sources, ingestion progress, and upgrades. See [Workspace settings](/docs/workspaces/settings). | The sidebar lists your Workspaces with their five most recent conversations. A pulse next to a Workspace shows that ctxpipe is ingesting content for it. Home shows the recent activity and commits of all Workspaces. Choose **Search** in the sidebar to find a Workspace or a page. ## How a Workspace stays current When a new commit arrives on the default branch, from ctxpipe or from a person, ctxpipe indexes it. See [How indexing works](/docs/workspaces/indexing). ## Read-only Workspaces ctxpipe writes to the Workspace repository only through the ctxpipe GitHub App. If it cannot push, the Workspace is **read-only**. For example, the repository is not on GitHub, or the App lost access. Search, chat, and the graph continue to work on the latest commit, and writes start again when access returns. **Settings** shows the reason. See [Write access](/docs/workspaces/settings#write-access). # How indexing works (https://docs.ctxpipe.ai/docs/workspaces/indexing) **Indexing** turns the files and the linked code of a Workspace into data that chat, search, the graph, and MCP can use. It runs in the background after each change, so new content can take a few minutes to show in answers. ## Two kinds of indexing ### The Workspace repository When the default branch of the Workspace repository changes, ctxpipe reads the knowledge files at that commit and rebuilds the index: units, links and claims for the graph, and embeddings for search. This step never changes files and calls no extraction model. The previous index serves until the new one is ready. ### Linked repositories When a linked repository changes, ctxpipe checks out its linked branch and builds code search and code navigation. Then it [extracts knowledge](#how-extraction-works) from the new commit and commits it to the Workspace repository, which starts the first kind of indexing. ## Changes and full rebuilds ctxpipe moves the search and graph data of each organization to a new knowledge store, one organization at a time. On the new knowledge store, an index run of the Workspace repository applies only the files that changed after the last indexed commit. A file that you remove from the repository leaves the index. Some events start a **full rebuild**. A full rebuild reads all the knowledge files at the newest commit again: * The last indexed commit is not in the Workspace repository any more. For example, a user force-pushed the default branch, and Git then deleted the old commits (a garbage collection). Without that commit, ctxpipe cannot find which files changed. * The first index run of a Workspace. * A new ctxpipe version that changes how ctxpipe reads files, a new embedding model, or a new root folder of the Workspace. * A change to the sources of the Workspace (`.ctxpipe/sources.yaml` or `sources/`). * Runs started for more than 10 different commits after the last indexed commit, the current commit included, and none of them completed. During a full rebuild: * The run takes longer than a usual run, because ctxpipe reads and embeds each file again. * Chat, search, the graph, and MCP continue to answer from the index. ctxpipe replaces the files in groups, so an answer during the run can include some files from the old commit and some from the new commit. * A file that is not at the newest commit leaves the index when the run ends. * When the run completes, the index is the same as the files at the newest commit. You do not have to do anything. If a run fails, the next run tries the same files again. ## How extraction works Extraction reads one linked repository at one commit. It writes services, apps, libraries, APIs, dependencies, data stores, infrastructure, decisions, and instructions as knowledge files. ### Prepare a sandbox ctxpipe clones the linked repository into an isolated sandbox. It adds a read-only copy of each Workspace repository that links the repository. See [Where extraction runs](#where-extraction-runs). ### Run the extraction skills Each step runs one skill. Scripts find the facts first, for example from manifests, lockfiles, and code patterns. A model confirms the facts, or adds what a script cannot see. Each step runs separately for each Workspace that links the repository. It updates or links the files that the Workspace already has, so that it does not add a second file for the same thing. The sandbox can only stage file changes. It cannot write to a repository. ### Check each file outside the sandbox ctxpipe checks each staged file before it accepts it. It rejects the file, and keeps the other files of the step, when: * the path is not in a knowledge folder of the [extracted files](/docs/workspaces/knowledge-files#extracted-files), or it is in `sources/`, `.ctxpipe/`, `.agents/`, a connector mirror, or `AGENTS.md`; * the change breaks the ownership rules, or changes a human section; * the front matter has an unknown key, or a claim has no `evidence` that exists in the linked repository at that commit; * a link does not resolve, or an external link goes to a repository that the Workspace does not know; * the text has control characters, hidden characters, raw HTML, or an HTML comment; * the file is larger than 64 KiB. A step can delete at most 20% of the files that its source owns. ### Publish one commit ctxpipe keeps the result of each step in a temporary ref, `refs/ctxpipe/ingest/…`, in the Workspace repository. ctxpipe does not index these refs. At the end of the run, ctxpipe commits the result to the default branch as one commit, `Update extracted knowledge`, and deletes the refs. Extraction publishes only to a Workspace at [format 3](/docs/workspaces/settings#repository-format) or later that ctxpipe can write to. Below format 3, the knowledge that the Workspace has stays readable. ## Guards against prompt injection and secrets The content of a linked repository can try to give orders to the model. The model gets the instruction to treat each file as data, never as instructions. ctxpipe also checks the result: * It rejects a file with a heading that addresses AI agents or models. * It rejects a file with an instruction-like phrase outside a quote. A quote of such a phrase must give a `Source:` path, and the text must be in that file of the linked repository. * It rejects all the files of a step when the text that the step writes has a value that looks like a secret. Examples are a private key and a connection string with a password. More examples are an `Authorization` header, and a key such as `*_TOKEN` with a real value. A known token format is also a secret. The scan also decodes base64 and percent-encoded text. The rejection gives the rule and the line, never the value. These guards limit what injected text can do. They cannot stop all of it. Review extracted files like other knowledge files. The `AGENTS.md` that ctxpipe writes tells coding agents to treat extracted files as data. ## Where extraction runs Extraction runs in a different sandbox from [Workspace chat](/docs/workspaces/chat#where-the-agent-runs). On hosted ctxpipe, each extraction run has its own Amazon Bedrock AgentCore sessions. The differences from chat: * **Closed network.** The sandbox can fetch the linked repository and the Workspace repositories of the run with git. It can call the ctxpipe model proxy and the ctxpipe code search tools. It can do nothing else. It cannot push, call the GitHub API, or reach another host or a private address. A public repository on another git host is reachable only while the sandbox clones it, before a model runs. * **No credential in the sandbox.** A ctxpipe proxy outside the sandbox adds each credential to a request on its way out. The GitHub tokens can only read, and ctxpipe revokes them when the run ends. * **Limited agent.** The model runs as a user with no admin rights. The user owns only the output folder `/ctxpipe/out` and the home folder `/ctxpipe/home`, so the model can write only there. It uses only the skills of ctxpipe. The `AGENTS.md` file and the skills in the linked repository do not change it. Extraction uses the sandbox slots of your organization. See [Sandbox limits](/docs/workspaces/chat#sandbox-limits). For self-hosted sandboxes, see [Chat sandboxes](/docs/self-hosting/configuration#chat-sandboxes). ## Re-ingestion When a new ctxpipe version changes what extraction writes, **Settings** offers the upgrade **Fresh re-ingestion of the source repositories**. It runs the extraction again on each linked repository. See [Upgrades](/docs/workspaces/settings#upgrades). ## Read the status **Settings** in each Workspace shows the hydrate status of the Workspace on the **Knowledge graph** card, and the status of each linked repository on the **Sources** card. A row of the **Sources** card that ingests now shows what ctxpipe does and how long it has run. A button on the row opens the stage and the progress of each run. In the sidebar, a pulse next to a Workspace shows that it ingests now. A re-ingestion that you start from an upgrade shows on the **Upgrades** card. For the status of a linked repository (**Pending**, **Indexing**, **Indexed**, **Indexed with issues**), see [Indexing status](/docs/workspaces/linked-repositories#indexing-status). For the hydrate status, see [Hydrate and skipped files](/docs/workspaces/settings#hydrate-and-skipped-files). When an index of the Workspace fails, the last good version continues to serve. Fix the cause, often front matter that is not valid or lost repository access, and push again. Indexing builds search and graph data from files and from claims that a model extracted. It does not prove an answer. Ask for the files, and check risky decisions against the current code. ## Three layers Content in a Workspace lives in three layers: 1. **Files**: the bytes in git, in the Workspace repository or a linked repository. 2. **Search**: the text and code index over those files. Chat and MCP can retrieve anything here. 3. **Graph**: typed nodes and relationships from links, claims, and `kind` front matter. Mirrored connector content becomes `Issue`, `Thread`, `PullRequest`, and similar nodes. It is never treated as instructions. Cross-tool links are joined by URL or identifier, not guessed. # Knowledge files (https://docs.ctxpipe.ai/docs/workspaces/knowledge-files) The knowledge of a Workspace is plain Markdown in its Workspace repository. ctxpipe keeps no hidden copy that can drift. Chat, search, and agents use what is on the default branch. ## One file, one unit Each Markdown file is one **knowledge unit**. Its path is its identity, so renaming a file creates a new unit. Keep units short and focused, and link to other units rather than copying them. On an empty repository, use `knowledge//.md`. On an existing repository, follow the folders already there. ```text workspace-repository/ ├── AGENTS.md # Workspace name + folder map ├── .agents/skills/ctxpipe-knowledge/ │ └── SKILL.md # How agents should write knowledge here ├── knowledge/ │ ├── services/checkout.md │ └── decisions/payments-provider.md ├── .ctxpipe/workspace.yaml # Repository format (host-only) ├── sources/ # One file for each source │ ├── github/acme/checkout.md │ └── linear/engineering.md ├── linear/ notion/ confluence/ # Connector mirrors └── slack/ pagerduty/ plain/ github/ ``` ## Relationships The knowledge graph has two layers. **Links.** A relative Markdown link from one file to another becomes a `LINKS_TO` relationship: ```markdown Checkout calls the [payments service](../services/payments.md). ``` **Claims.** Optional `claims:` front matter states typed facts about the unit: ```markdown --- kind: Service claims: - to: ../services/payments.md predicate: DEPENDS_ON confidence: 0.7 valid_from: 2026-03-01 --- # Checkout ``` | Field | Meaning | | ------------------------ | -------------------------------------------------------------------------------------------------- | | `to` | Required. File-relative path of the related unit. Items without `to` are skipped. | | `predicate` | The relationship, for example `DEPENDS_ON`, `PART_OF`, `OWNS`, `SUPERSEDES`. | | `confidence` | How sure the author is: about `0.5` typical, `0.7` strong, `0.85` and above rare. | | `valid_from`, `valid_to` | When the fact holds. A missing `valid_from` defaults to the time of the commit that introduced it. | | `source` | Optional pointer to where the fact came from. | `kind` is optional and sets the unit's node type in the graph (`Service`, `API`, `Library`, `Decision`, `Pattern`, and so on). Without it, the unit appears as a generic knowledge unit. Older facts count for less over time. Recent, well-sourced facts rank highest in answers. ## Skipped files ctxpipe skips a file that has one of the errors below, and continues with the other files. One bad file does not stop the Workspace. A skipped file stays in **Files**, because it is in git, but chat, MCP, search, and the graph do not use it. The **Knowledge graph** card in **Settings** lists the skipped files of the latest commit under **ctxpipe skipped N files**. | Reason | Cause | Fix | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------- | | Front matter is not valid | The front matter has no closing `---` line, or its YAML is not valid. | Correct the front matter. | | Repeats a linked repository or the Workspace's own repository | Shows only in results from an earlier version, for a `repositories/*.md` file. ctxpipe no longer reads these files. | None. | Push the fix to the default branch. When ctxpipe reads the new commit, the file is a knowledge unit again, and it leaves the list. Other errors stop the read of the whole commit. For example, a set of files that is too large shows **The knowledge graph did not load** in **Settings**. See [Workspace settings](/docs/workspaces/settings#hydrate-and-skipped-files). ## Sources and linked repositories Each file under `sources/` is a source. The GitHub source files link the code repositories of a Workspace. See [Linked repositories](/docs/workspaces/linked-repositories) and [Workspace settings](/docs/workspaces/settings#sources). ctxpipe does not read files in `repositories/`. When an organization moves to Workspaces, the upgrade pull request turns those files into source files. ## Extracted files ctxpipe writes the knowledge that it [extracts from linked repositories](/docs/workspaces/indexing#how-extraction-works) only to these folders: `knowledge/codebases/`, `services/`, `apps/`, `libraries/`, `apis/`, `operations/`, `databases/`, `infrastructure/`, `streams/`, `decisions/`, `teams/`, and `instructions/`. ctxpipe sets these front matter fields. Do not change them by hand: | Field | Meaning | | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `generated_by` | `ctxpipe-extract`. ctxpipe extracted the file. | | `sources` | The source files that own the file, for example `sources/github/acme/checkout.md`. | | `source_sha` | The commit of the linked repository that ctxpipe read. | | `root` | The folder of the package in the linked repository. It is `.` for the whole repository. Only `Repository`, `Service`, `App`, and `Library` files have it. | | `skill`, `bundle` | The extraction step that wrote the file, and the version of the extraction. | Each claim of an extracted file has `evidence`: a path in the linked repository, and `script` or `model`. ctxpipe sets `confidence` from it: `0.7` for `script` and `0.5` for `model`. To change an extracted file: * Put your own text between `` and ``. Ingestion never changes it. * To take over the file, remove `generated_by`. After that, ingestion can only add claims and new lines in its `## Related` section. A file that a source owns alone can change fully on the next run. When you remove a repository source, ctxpipe deletes the files that only that source owned. ctxpipe also removes the source from the files that it shares with other sources. A file with a human section stays. ctxpipe removes only its own fields and claims. For all the commits that ctxpipe makes, see [What ctxpipe writes](/docs/workspaces#what-ctxpipe-writes). # Linked repositories (https://docs.ctxpipe.ai/docs/workspaces/linked-repositories) **Linked repositories** are the code repositories that a Workspace searches. ctxpipe indexes each one for code search, and extracts its services, APIs, libraries, and their relationships into the Workspace repository as knowledge files. A repository can be linked to many Workspaces. Each Workspace keeps its own index, so a search in one Workspace never returns the code of another. ## Link repositories Each linked repository is one source file in the Workspace repository, for example `sources/github//.md`. A pull request adds or removes the file. The change applies after you merge the pull request. ### Open Add repositories In the Workspace, open the **Settings** tab. Under **Sources**, choose **Add repositories** in the **GitHub** group. ### Choose the repositories * **Select GitHub**: search the repositories that the ctxpipe GitHub App can see, and select one or more. Only an organization owner or admin can list them. * **Paste URL**: type the URL of a public repository on any git host, or of a public GitHub repository with no GitHub connection. See [Plain git repositories](#plain-git-repositories). Choose **Open a pull request**. One pull request can link up to 20 repositories. ### Merge, then wait for indexing Merge the pull request. Until then, the row shows **Awaiting merge**. After the merge, ctxpipe queues each repository for indexing. To remove a repository, choose the unlink button on its row. ctxpipe opens a pull request that deletes the source file. After the merge, ctxpipe stops searching the repository and removes the knowledge that it extracted from it. ## Pin a branch or tag A source reads the default branch of the repository. To read another branch or a tag, add `ref` to the front matter with a pull request: ```markdown --- provider: github connection: con_example repository: acme/payments ref: release --- The payments API. Use it for questions about refunds and card tokens. ``` The body is an optional description of what the repository is for. People own the body, and ctxpipe never changes it. You can rename or move the file inside `sources/github/`. The pin stays with the repository. ## Plain git repositories ctxpipe can read a repository that needs no credential: a public repository on another git host, or a public GitHub repository. Use **Paste URL**, or add a source file under `sources/git/` with a pull request: ```markdown --- provider: git url: https://git.example.org/team/tools.git --- ``` ctxpipe checks the URL before it opens the pull request. On hosted ctxpipe, the URL must: * use `https://` and the default port; * have a public domain name as its host, not an IP address or a name such as `localhost`, and resolve only to public addresses; * have no user name, password, query, or fragment. ctxpipe does not support ssh (`git@…`, `ssh://`), `git://`, or `file://` URLs, and does not follow a redirect. Use the exact clone URL. A repository that needs a credential works only on GitHub, through the GitHub App. A self-hosted deployment can also read a git server inside your network. See [Plain git repositories](/docs/self-hosting/configuration#plain-git-repositories) in the self-hosting configuration. ## Indexing status | Status | Meaning | | ----------------------- | --------------------------------------------------------------------------------------------------------------- | | **Pending** | Linked, waiting for indexing. | | **Indexing** | ctxpipe is building search and extracting knowledge. Answers can be incomplete. | | **Indexed** | Search and extraction are done for the current commit. | | **Indexed with issues** | Search works, but part of the index failed, usually code navigation for one language. The row shows the reason. | Indexing runs again when the linked branch or tag changes. The row of the repository in the **Sources** card of **Settings** shows the progress of each run. ## What indexing builds * **Code search**: text, file, and symbol search over the linked branch. * **Code navigation**: definitions and references for the supported languages. * **Extracted knowledge**: services, APIs, libraries, infrastructure, and instructions such as `AGENTS.md`, skills, and ADRs, as knowledge files in the Workspace repository. See [How extraction works](/docs/workspaces/indexing#how-extraction-works). ctxpipe only reads a linked repository. It never writes to it. See [Code repositories stay read-only](/docs/workspaces#code-repositories-stay-read-only). ## Issues and pull requests For a linked GitHub repository, ctxpipe also mirrors its issues and its closed pull requests into `github/issues/` and `github/pulls/` in the Workspace repository. When you unlink, new items stop, and the files already mirrored stay. See [GitHub connector](/docs/connections/source-connectors/github). # Workspace settings (https://docs.ctxpipe.ai/docs/workspaces/settings) Open a Workspace and choose the **Settings** tab. The pane has a summary line at the top, and then one card for each part of the Workspace. The summary line names the cards that need you, and the cards that work now. Each name is a link to its card. The line has one of these forms: * **Everything is up to date.** * **N parts need you:** (or **1 part needs you:**) and the names of the cards. The line can also say **Running:** and the names of the cards that work now. * **Nothing needs you. Running:** and the names of the cards that work now. A card that needs you has an amber edge. These are the cards, in the order of the pane: | Card | What it holds | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Upgrades** | The upgrades that wait or run, the automatic-upgrades switch, the repository format, and the history. See [Upgrades](#upgrades). | | **Sources** | The code repositories and tools that the Workspace reads, and the live work of each one. See [Sources](#sources). | | **Knowledge graph** | Whether ctxpipe loaded the graph, and the files that ctxpipe skipped. See [Hydrate and skipped files](#hydrate-and-skipped-files). | | **Repository** | The Workspace repository, which this page calls the **Context repository**, the write access, and the **Root folder**. See [Repositories](#repositories) and [Write access](#write-access). | | **Workspace** | The **Name** (stored in `AGENTS.md`) and the **URL name**. If you change the URL name, old links stop working. Choose **Save** to keep a change. | | **Danger zone** | **Delete Workspace**. This removes the Workspace and its conversations from ctxpipe. Your repository and its files stay. | Each member can read the pane. Only an organization owner or admin can start an upgrade or change a source. ## Upgrades On the **Upgrades** card, each upgrade that needs you or runs now is a row. The dot of the row shows its kind: amber for an item that needs you, teal for work that runs, and red for a failure. The card header shows **Up to date** when no row is open and the format is the latest. It shows **Checking the format** while ctxpipe reads the format file. A re-ingestion row also shows its spend so far. You can leave the page. The work continues. Applied upgrades show under **History**. Home shows an alert for each Workspace that has an upgrade to finish. **Review upgrade** on the alert opens this card. An owner or admin chooses **Start** on a row. A member sees **Ask an admin** in place of the button. To stop a re-ingestion that is queued or runs, an owner or admin chooses **Cancel** on its row, then confirms. ctxpipe stops the run and each ingestion run of it. The run makes no more model calls and publishes nothing more. A repository that finished before the cancel keeps its new knowledge. The row shows **Canceled**, and ctxpipe does not start it again by itself. A format upgrade has no **Cancel**: it only opens a pull request. Upgrades move the Workspace repository to the layout that this version of ctxpipe reads. Nothing changes until you merge the pull request. There are two kinds of upgrade: * **Upgrade to format N** changes the layout of the repository. ctxpipe opens one pull request on the branch `ctxpipe/upgrade/format-`, with one commit for each step. The last commit writes `.ctxpipe/workspace.yaml`. Read the notes under the row before you merge. * **Fresh re-ingestion of the source repositories** reads each linked repository again and publishes the new knowledge. See [How extraction works](/docs/workspaces/indexing#how-extraction-works). It needs format 3. For the cost and the restart rules, see [Upgrade a Workspace](/docs/self-hosting/upgrades#upgrade-a-workspace). | Status | What to do | | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | **Ready to start** | An owner or admin chooses **Start**. | | **Queued**, **Preparing the pull request**, **Waiting** | Wait. | | **Waiting for merge** | Review and merge the pull request. If the card does not change after a few minutes, choose **Check again**. | | **Applied** | Nothing. The upgrade is done. | | **Failed** | Read the reason, fix the cause, then choose **Start again**. | | **Canceled** | An owner or admin stopped the re-ingestion. To run it, choose **Start again**. | | **Blocked** | Read the reason and remove the cause. See the table below. | If someone closes the pull request without a merge, the upgrade goes back to **Ready to start**. | Blocked title | Cause | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ctxpipe cannot open the upgrade pull request | ctxpipe has no write access. See [Write access](#write-access). | | ctxpipe cannot publish the re-ingestion | ctxpipe has no write access. See [Write access](#write-access). | | Apply the format upgrade first | The repository is below format 3. Merge the format upgrade, then choose **Check again**. The item then shows **Waiting** until ctxpipe reads the new format. | | This deployment cannot run the re-ingestion | The deployment has no ingest sandbox. The operator must configure one. | ### Automatic upgrades By default, nothing starts by itself. To change this, an owner or admin turns the **Automatic upgrades** switch on or off, or chooses **Always update** on the Home alert. ctxpipe then commits the setting into `.ctxpipe/workspace.yaml` on the default branch. When automatic upgrades are on, ctxpipe starts each upgrade when it finds it. A merge always needs a person. ### Repository format Each Workspace repository holds `.ctxpipe/workspace.yaml` below its root folder. The key `format` is the version of the repository layout. The optional key `upgrades` is `manual` (the default) or `automatic`. ```yaml format: 3 upgrades: manual ``` The format line of the **Upgrades** card compares this format with the format that ctxpipe reads. | Format | Layout | | ------ | --------------------------------------------------------------------------------------------------------------------- | | 0 | No format file: a repository from before Workspaces. | | 1 | The knowledge that a fresh ingestion cannot rebuild is exported. Sources are in the older `.ctxpipe/sources.yaml`. | | 2 | One Markdown file for each source under `sources/`. Source changes need format 2. | | 3 | Ingestion writes ownership front matter in each knowledge file. This is the current format. Ingestion needs format 3. | Only the setup commit of a new Workspace, an upgrade pull request, and the automatic-upgrades setting write this file. A format that this version of ctxpipe does not know, or a file that does not parse, makes the Workspace **Read-only**. Reads continue. ## Sources A source tells the Workspace what to read through a connection. Each source is one Markdown file at `sources//.md` in the Workspace repository. The front matter holds the provider, the connection, and the scope. The body is an optional description that people and agents read. The path of the file is the identity of the source. The file on the default branch is the live scope. Each change to a source is a pull request that a person merges. No agent can change `sources/` without that review. The buttons show only when ctxpipe can open a pull request. See [Write access](#write-access). The **Sources** card has one group for each provider that has a connection or a source. Each row shows the source, its file, and its status. **Add another connection** opens **Connectors**. While ctxpipe works on a source, its row shows what ctxpipe does, for example **Syncing** or **Publishing**, and how long it has run. A row that has more than one run has a button that opens the stage and the progress of each run. You can leave the page. The work continues. While ingestion runs, the sidebar shows a pulse next to the Workspace. ### Add, edit, rename, or remove a source | Action | How | | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Add | Choose the add button in the group of the provider (**Add repositories** for GitHub). Select the scope, then choose **Open a pull request**. | | Edit | Open the actions menu of the row, then choose **Edit**. Change the scope, then choose **Open a pull request**. | | Rename | Open the actions menu of the row, then choose **Rename**. ctxpipe moves the file, and updates the knowledge files that name the old path, in the same pull request. | | Remove | Open the actions menu of the row, then choose **Remove**. After the merge, the next sync deletes the files that the source wrote. Git history keeps them. Captured Slack threads stay. | In the add dialog, **File name** is optional. Without a name, ctxpipe names the file from the provider, for example `linear` or `linear-2`. A GitHub source is always `sources/github//.md`. Only an organization owner or admin can list the content of most providers. A member sees **Ask an admin to choose what this source syncs**. Any member can list Confluence spaces. A row with an open pull request shows **Awaiting merge**, **Rename awaiting merge**, or **Removal awaiting merge**, with a link to the pull request. After the merge, the row updates in a few seconds. ### Source status | Status | Meaning | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | **Connected** | The file, the connection, and the scope work. The source syncs. The Workspace creation form shows **OK** for this status. | | **Not valid** | The file does not parse, has a wrong shape, or breaks a rule. The sync skips it. The reason names the problem. | | **Connection missing** | The connection of the source was removed. The file stays. | | **Out of access** | The connection cannot reach the scope, for example a repository that is not in the GitHub App installation. ctxpipe checks again later. | These rules apply across files: * Two sources cannot name the same GitHub repository, or write to the same folder. The second file is **Not valid**. * A GitHub source cannot name the Workspace repository. * A source file larger than 64 KiB, or a file that is not text, is **Not valid**. ### Source file format Each file has the front matter key `provider`. Each file except a plain git source also has `connection`. An unknown key makes the file **Not valid**. | Provider | Path | Scope keys | | ---------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | GitHub | `sources/github//.md` | `repository` (`owner/name`), optional `ref` (a branch or a tag) | | Linear | `sources/linear/.md` | `teams` (team keys), `projects` (project ids), optional `customerRequests` (`exclude` or `limited`). At least one team or project. | | Notion | `sources/notion/.md` | `pages` and `databases`, each a list of `id` and `title`. At least one of them. | | Confluence | `sources/confluence/.md` | `spaces`: a list of `key` and `pages` (`all` or a list of page ids) | | Slack | `sources/slack/.md` | `channels`: a list of channel ids | | PagerDuty | `sources/pagerduty/.md` | `services`: a list of service ids | | Buildkite | `sources/buildkite/.md` | `pipelines`: a list of pipeline slugs | | Plain | `sources/plain/.md` | `helpCenters` (Help Center ids) and `labels` (label type ids). At least one of them. | | Plain git | `sources/git//.md` | `url` only | Each path segment after the provider folder matches `[a-z0-9][a-z0-9._-]{0,99}`. The keys `id` and `kind` are not allowed. ## Hydrate and skipped files Hydrate is the step that reads the files of the repository at one commit and builds the search index and the graph. The **Knowledge graph** card shows its status: | The card shows | Meaning | | ------------------------------------------ | ----------------------------------------------------------------------- | | **The knowledge graph is loaded.** | ctxpipe read the latest commit. | | **The knowledge graph waits to load.** | ctxpipe did not start yet. | | **ctxpipe loads the knowledge graph now.** | ctxpipe reads a commit now. The previous index serves until it is done. | | **The knowledge graph did not load** | ctxpipe could not read the repository. The card shows the error. | While ctxpipe indexes a new commit, the card also shows **Indexing the newest commit…** and the time since it started. Before the first hydrate ends, the Workspace page shows **Preparing** and the name of the Workspace. If the first read fails, it shows **Could not prepare the Workspace** with the reason and **Try again**. When ctxpipe skips files, the card shows **ctxpipe skipped N files**, with the path and the reason for each file. Chat, MCP, search, and the graph do not use a skipped file. Select a path to open the file in **Files**. Fix the file and push the fix to the default branch. See [Skipped files](/docs/workspaces/knowledge-files#skipped-files). ## Repositories * **Context repository**: choose **Change** to move the Workspace to a different repository. The dialog has the same options as [Create a Workspace](/docs/workspaces/create-workspace). The Workspace and its conversations stay. The files in the old repository stay where they are. Until the new repository is loaded, answers come from the old one. * **Root folder**: the folder of the repository that holds the Workspace. See [Root folder](/docs/workspaces/create-workspace#root-folder). ## Write access ctxpipe writes to the Workspace repository through the ctxpipe GitHub App. The **Repository** card shows the result of the last check in one line. The header of a conversation also shows a tag, but not when ctxpipe can push. The tag has the same title as the line. | The UI shows | Meaning | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ctxpipe can push to this repository. | The GitHub App can push to the repository. The state is writable. | | **Changes by pull request** | The default branch is protected. Conversations and source changes use pull requests. Connector mirrors and extracted knowledge cannot push to the default branch, so they stop. | | **Checking write access** | ctxpipe is checking. The pane checks again by itself. On the card, the line says "ctxpipe checks write access now." If the check takes longer than usual, the card shows **Still checking write access**. | | **Could not check write access** | GitHub returned an error or a rate limit. Choose **Check again**. | | **Read-only** | ctxpipe cannot write. Search, chat, and the graph continue to work on the latest commit. | A **Read-only** Workspace shows the reason: | Reason | What to do | | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | The remote is not GitHub. | Select a GitHub repository as the context repository. | | GitHub is not connected for this organization. | An owner or admin connects GitHub on **Connectors**. Then choose **Select the repository again**. | | The repository is not in the GitHub App installation. | Add the repository to the installation in GitHub. | | The GitHub App cannot write to this repository. | Give the GitHub App **Contents: Read and write**. | | The Workspace needs a newer ctxpipe version. | The repository format is newer than this ctxpipe version. Upgrade ctxpipe. See [Repository format](#repository-format). | | The file `.ctxpipe/workspace.yaml` is not valid. | Repair the file in the repository. | After you fix the cause, reload the page. ctxpipe checks again at most once a minute. # Authentication (https://docs.ctxpipe.ai/docs/self-hosting/authentication) ctxpipe uses [Better Auth](https://www.better-auth.com) for identity management. All browser sign-in, provider callbacks, sessions, and MCP OAuth go through the same public backend origin. ## Required auth boundary | Variable | Requirement | | ---------------------- | ------------------------------------------------------------------------------------------------------------- | | `AUTH_SECRET` | A stable random value of at least 32 characters. Generate with `openssl rand -hex 32`. | | `AUTH_BASE_URL` | The public HTTPS backend origin users and providers can reach. | | `AUTH_ALLOWED_ORIGINS` | Optional comma-separated extra browser origins. Keep this list narrow. | | `SMTP_CONNECTION_URL` | Required for email verification, password reset, and invitation email delivery, unless `EMAIL_TRANSPORT=ses`. | | `EMAIL_FROM_ADDRESS` | The sender address for transactional email. | `AUTH_SECRET` protects sessions and encrypted auth material. Do not generate a new value on every deploy. A rotation can invalidate sessions. It can also prevent ctxpipe from reading the encrypted signing keys that you have. Plan each rotation as an operational change. Open the product through `AUTH_BASE_URL`. Do not send browser users directly to the private UI service. Provider callbacks, MCP OAuth, cookies, and the API need one consistent origin. ## Email and password Email/password sign-in is available by default. Set up SMTP before you invite a team. Then verification, invitation, and password-reset messages leave the deployment reliably. When you set SMTP and `EMAIL_FROM_ADDRESS`, a new email/password account must verify its address before it can sign in. ctxpipe sends a fresh link to an unverified account that signs in. A password reset also verifies the address. It signs out the other sessions of the account. Without SMTP, ctxpipe only logs the messages. Verification is then not required, and the backend logs a warning at startup. ## Social providers (OAuth) ctxpipe supports GitHub, Google, and Microsoft social sign-in. Each provider is on only when you set its client ID and secret in the backend environment. Then the sign-in option shows. | Provider | Env vars required | | --------- | ------------------------------------------------ | | GitHub | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` | | Google | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` | | Microsoft | `MICROSOFT_CLIENT_ID`, `MICROSOFT_CLIENT_SECRET` | GitHub social sign-in credentials are separate from the [GitHub App](/docs/self-hosting/github) that your deployment owns. That App gives access to repositories and writes connector content. Register each provider callback under `AUTH_BASE_URL`. Test sign-in through the same public origin that users use in production. ## Passkeys and two-factor authentication Passkeys are available on compatible browsers and platforms. Users can also enable TOTP two-factor authentication with a standard authenticator app. Verify both through your production TLS domain. WebAuthn credentials belong to one origin. ## MCP OAuth and API keys ctxpipe is an OAuth provider for downstream MCP clients. It supports discovery, dynamic client registration, and device authorization for compatible headless clients. Give the client this URL: ```text {AUTH_BASE_URL}/mcp ``` Each OAuth grant holds a set of organizations. The user selects the organizations on the consent screen, and the user can change the set later on the **Connected apps** page (`/.auth/account/connected-apps`). Each tool call names one organization with the `org` argument. The server allows the call only when the organization is in the grant set and the user is still a member of it. For the old `?orgSlug=` URL parameter, see [MCP](/docs/self-hosting/mcp). OAuth is the normal MCP flow for people. The consent screen grants `ctxpipe:read` and `offline_access`. It also offers **Also allow changes** (`ctxpipe:write`) when a client asks for it. The MCP tools need only `ctxpipe:read`. API keys never hold `ctxpipe:write`. | Token | Lifetime | | --------------------------------------- | ---------------------------------------------------------- | | Browser session | 30 days | | OAuth access token | 4 hours, renewed with the refresh token (`offline_access`) | | OAuth access token with `ctxpipe:write` | 1 hour | API keys are stored in the database of this deployment, so there is no operator secret to configure. For personal and organization keys, the headers, and the client setup, see [API keys](/docs/mcp/api-keys). When discovery or authorization fails, see [MCP diagnostics](/docs/self-hosting/mcp). ## Production checks * `AUTH_SECRET` is stable across backend instances and deploys. * `AUTH_BASE_URL` exactly matches the browser origin. * Provider callbacks use HTTPS and the same host. * Extra trusted origins are explicit rather than wildcarded. * SMTP can deliver invitation and reset messages. * Passkey registration and TOTP sign-in work on the production domain. * An MCP client can discover OAuth and complete authorization. # Configuration (https://docs.ctxpipe.ai/docs/self-hosting/configuration) Environment variables control the configuration. Set only the values that your deployment needs. Treat the required auth, database, model, and public-origin values as production secrets. ## What a deployment needs * A [deployment path](/docs/self-hosting#choose-a-deployment-path): Docker Compose or AWS CDK. * PostgreSQL with pgvector. Compose uses PostgreSQL 17. AWS CDK uses Aurora PostgreSQL 16. * A graph database. FalkorDB is in the Compose stack. * ClickHouse 26.10 for the new knowledge store. ClickHouse is in the Compose stack. AWS CDK does not deploy it yet. * A model provider: OpenRouter, Azure OpenAI, Amazon Bedrock, or another OpenAI-compatible API. * A public HTTPS origin for auth callbacks, webhooks, MCP, and Forge. * Optional: an SMTP provider for email. On Docker Compose, set the values in `.env`. Compose passes only some of them to the services. See [Settings that need an override file](/docs/self-hosting/deployment/docker#settings-that-need-an-override-file). ## Required variables The backend does not start without these: | Variable | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DATABASE_URL` | Runtime PostgreSQL URL for backend, worker, and codesearch. Use the `ctxpipe_app` role, which has no `BYPASSRLS`. Example: `postgresql://ctxpipe_app:pass@host:5432/db`. `migrate` uses the table-owner URL. Compose and AWS CDK set both URLs. | | `AUTH_SECRET` | A random string of 32 or more characters, for session signing. Generate it with `openssl rand -hex 32`. Keep it stable. A new value signs everybody out and, without `CONNECTION_SECRETS_ENCRYPTION_KEY`, makes the stored connector secrets unreadable. | Set these too for a deployment that people use: | Variable | Description | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `AUTH_BASE_URL` | The public URL of your deployment, for example `https://ctx.example.com`. The default, `https://localhost:3000`, works only on your machine. | | `CTXPIPE_PUBLIC_APP_URL` | The public backend URL in the UI image, at build time. Usually the same as `AUTH_BASE_URL`. Compose and AWS CDK read it. | | `MODEL_PROVIDER_API_KEY` | The API key of your model provider. Required for each provider except `bedrock`. | | `CODESEARCH_URL` | The URL of codesearch. Repository search and indexing need it. The backend and the worker must both reach it. Compose and AWS CDK set it. | ## Email Email is optional. Without both values, ctxpipe turns off email verification and sends no password reset or invitation email. The backend logs a warning. | Variable | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SMTP_CONNECTION_URL` | SMTP connection URL, for example `smtps://user:pass@smtp.resend.com:465` | | `EMAIL_FROM_ADDRESS` | From address for transactional email, for example `noreply@example.com` | | `EMAIL_TRANSPORT` | `smtp` (the default) or `ses`. With `ses`, the backend sends through the Amazon SES v2 API. It uses the AWS credentials of its process (for example an ECS task role) and its `AWS_REGION`. It needs no `SMTP_CONNECTION_URL`. The role needs `ses:SendEmail` on the sender identity. | ## Social sign-in All social providers are optional. Omit the variables to disable that provider. | Variable | Description | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GITHUB_CLIENT_ID` | Client ID of the GitHub App (sign-in, and the check that a user can access an installation) | | `GITHUB_CLIENT_SECRET` | Client secret of the GitHub App | | `GITHUB_WEB_URL` | Optional. The base URL, in place of `https://github.com`, of the GitHub pages that the browser opens for the deployment's App (the authorize page and the install page), when a forwarder serves them. Set it together with `GITHUB_REDIRECT_URI`. Default: `https://github.com` | | `GITHUB_REDIRECT_URI` | Optional. The callback URL that the GitHub App lists, when a forwarder serves it. Set it together with `GITHUB_WEB_URL`. Default: `{AUTH_BASE_URL}/.auth/api/v1/auth/callback/github` | | `GOOGLE_CLIENT_ID` | Google OAuth client ID | | `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | | `MICROSOFT_CLIENT_ID` | Microsoft Entra application client ID | | `MICROSOFT_CLIENT_SECRET` | Microsoft Entra client secret | See [Authentication](/docs/self-hosting/authentication) for OAuth and session behavior. ## Models and embeddings | Variable | Description | | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `MODEL_PROVIDER` | Optional. `openai-like` default, `openrouter`, `azure`, or `bedrock`. | | `MODEL_PROVIDER_API_KEY` | API key for OpenAI-compatible providers. Omit for native Bedrock IAM. | | `MODEL_PROVIDER_URL` | Optional for `openai-like` and `openrouter`; required for Azure; not used by native Bedrock. | | `MODEL_BEDROCK_AWS_REGION` | Optional explicit region for native Bedrock; otherwise use the AWS runtime region. | | `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, `MODEL_HIGH_NAME` | Optional model names per tier. | | `MODEL_EMBEDDING_NAME` | Optional embedding model ID. | | `MODEL_VISION_NAME` | Optional. The one OpenRouter model that reads images for the knowledge plane (default `openai/gpt-6.1-terra`). It must take image input and have a ZDR endpoint on OpenRouter. | See [Model configuration](/docs/self-hosting/models) for provider-specific examples and embedding requirements. ## Graph database | Variable | Description | | ------------------------------------ | ------------------------------------------------------------------------------------------------ | | `GRAPH_DB_PROVIDER` | Optional. `falkordb` (default), `neo4j-enterprise`, `neo4j-community`, `memgraph`, or `neptune`. | | `GRAPH_DB_URI` | FalkorDB Redis URI or Bolt URI for another supported graph database. | | `GRAPH_DB_USER`, `GRAPH_DB_PASSWORD` | Optional credentials for providers that require them. | See [Graph databases](/docs/self-hosting/graph-databases) for provider support and tenancy behavior. ## Knowledge store ClickHouse holds the content of the new knowledge store: the documents, their sections, and the relations between them. ctxpipe makes this content from the files of each Workspace repository. The new knowledge store is off for every organization. A later release explains how to turn it on. | Variable | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CLICKHOUSE_URL` | Required for an organization on the new knowledge store. The ClickHouse HTTP URL with the user, the password, and the database: `http://user:password@host:8123/database`. URL-encode special characters in the password. Each process gets a different user (see below). | | `CLICKHOUSE_READER_PASSWORD` | Migrations only. The password of the user `ctxpipe_reader`. | | `CLICKHOUSE_WRITER_PASSWORD` | Migrations only. The password of the user `ctxpipe_writer`. | Give each process its own ClickHouse user in `CLICKHOUSE_URL`: * **Migrations:** an admin user with SQL access management (`access_management`). The migrations create the tables, a row policy for each table, and the two users below. Give the admin user to the migrations only. * **Backend:** `ctxpipe_reader`. It can only read the knowledge tables. * **Worker:** `ctxpipe_writer`. It can read, insert, and delete rows in the knowledge tables. It cannot manage users, grants, or row policies. The migrations create or update a user only when its password is set. Without the password, the user does not exist and cannot connect. The migrations refuse the same value for two of the admin, reader, and writer passwords. They also refuse the reader and writer passwords when the admin password is the Compose default `ctxpipe`. They store a salted SHA-256 hash of the password. To change a password, set the new value and run the migrations again. If `CLICKHOUSE_URL` is not set, the migrations skip ClickHouse. The knowledge tables move a part to the volume `cold` when its newest row is 30 days old. Thus the `default` storage policy of the server must have the volumes `hot` and `cold`, or the migrations fail. Compose loads [`clickhouse-storage.xml`](https://github.com/ctxpipe-ai/ctxpipe/blob/main/apps/backend/src/knowledge-store/clickhouse-storage.xml) into `/etc/clickhouse-server/config.d/`. On another ClickHouse server, add that file in the same folder. The Compose service keeps both volumes on its local disk. ### Keep old parts on S3 This example replaces the disk of the volume `cold` with an S3 bucket behind a cache disk. Change the bucket URL, the Region, and the cache size: ```xml title="clickhouse-storage-s3.xml" s3 https://example-bucket.s3.us-west-2.amazonaws.com/clickhouse/ true cache cold_s3 cold 50Gi 131072 default cold 0 0 ``` On Compose, mount your file in place of the default file, in a `docker-compose.override.yml`: ```yaml title="docker-compose.override.yml" services: clickhouse: volumes: - ./clickhouse-storage-s3.xml:/etc/clickhouse-server/config.d/storage.xml:ro environment: AWS_ACCESS_KEY_ID: ${CLICKHOUSE_S3_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${CLICKHOUSE_S3_SECRET_ACCESS_KEY} ``` The search reads small ranges of the vector column, so the example sets `boundary_alignment` of the cache disk to `131072` (128 KiB). With the default alignment of 4 MiB, a search that is not in the cache downloads about 1 GiB from S3 for 46 MiB of data on 1 million sections. The cache files go to `/var/lib/clickhouse/caches/cold/`. Do not put the cache path under `/var/lib/clickhouse/disks/`: the image then makes that folder as `root`, and ClickHouse cannot start. With `use_environment_credentials`, ClickHouse uses the AWS credentials of its environment, or the instance role on EC2. Give them read, write, list, and delete access to the bucket prefix only. Set the S3 disk before the first start. A part that is already on the local volume `cold` does not move to S3. ### Recover the knowledge store ClickHouse holds only content that ctxpipe makes from the Workspace repositories. If you lose the ClickHouse data: 1. Restore a ClickHouse backup, if you have one. 2. Run the migrations again, also after a restore. They make each missing table and the users `ctxpipe_reader` and `ctxpipe_writer`. On Compose, set the reader and writer passwords in `.env`, then run: ```bash docker compose --profile deploy run --rm migrate ``` 3. Delete the index state of the knowledge store in Postgres. Do this also after a restore: the backup can be older than the index state, and the next run then does not apply the commits between them. ```bash docker compose exec postgres psql -U ctxpipe -c "DELETE FROM knowledge_state" ``` The next hydrate of each Workspace is a full rebuild. Until a Workspace completes its full rebuild, the graph database answers for it. ## Chat sandboxes Workspace chat runs each conversation's agent in a sandbox. Docker Compose and AWS CDK set these values for you. On other platforms, set them on the backend and the worker. | Variable | Description | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SANDBOX_PROVIDER` | `docker`, `agentcore`, or `unsandboxed`. Set it to `docker` on a self-hosted deployment; Docker Compose and AWS CDK set it. Without a value, chat uses Docker, but the plain git rules then use the strict hosted rule: they use the self-host rule only when the value is exactly `docker` (see below). If the Docker daemon does not answer, chat returns 503. The backend and worker images refuse `unsandboxed`: it is only for local development. | | `DOCKER_HOST` | Docker API address, for example `tcp://dind:2376`. If you do not set it, the backend uses the local Docker socket. | | `DOCKER_TLS_VERIFY`, `DOCKER_CERT_PATH` | Mutual TLS to the Docker API. Set `DOCKER_TLS_VERIFY=1`. `DOCKER_CERT_PATH` is a directory with `ca.pem`, `cert.pem`, and `key.pem`. | | `SANDBOX_CHAT_IMAGE` | Chat sandbox image on that daemon. The default is `ctxpipe-chat-sandbox:opencode-`, which is in no registry. On other platforms, build `scripts/chat-sandbox` on that daemon, or use `ghcr.io/ctxpipe-ai/chat-sandbox:` once that package is public. Compose builds `ctxpipe-chat-sandbox:local`. AWS CDK uses `ghcr.io/ctxpipe-ai/chat-sandbox` at the commit SHA tag that the package pins. | | `SANDBOX_CALLBACK_HOST` | Optional. Host name that sandboxes use to send requests to the backend, through Agent Vault. It must be a host name with a dot (for example `backend.sandbox.ctxpipe.internal`), not an IP address: Agent Vault rules take host names. If you do not set it and `DOCKER_HOST` is remote, the backend uses its own address on the route to the Docker daemon, which works only with `SANDBOX_CALLBACK_DNS_SUFFIX`. Compose sets it. | | `SANDBOX_CALLBACK_DNS_SUFFIX` | Optional. With it, that derived address becomes the name `ip----.`, for example the VPC DNS name on AWS. AWS CDK sets it. | | `AGENT_VAULT_ADDR` | Required for Docker sandboxes. The Agent Vault API, for example `http://agent-vault:14321`. Without it, or when Agent Vault does not answer, Docker chat returns 503. Compose and AWS CDK set it. | | `AGENT_VAULT_OWNER_PASSWORD`, `AGENT_VAULT_OWNER_PASSWORD_FILE` | The Agent Vault owner password, as a value or as a file. Compose and AWS CDK generate it; do not set it there. On other platforms, generate one and keep it: the backend registers it on its first login, and an Agent Vault with an owner refuses another password. | | `AGENT_VAULT_SANDBOX_PROXY_HOST` | Optional. The host name that sandboxes use to connect to the Agent Vault proxy, when sandboxes cannot use the host of `AGENT_VAULT_ADDR`. Give only a host name: no scheme, no port, and no path. The port is the proxy port that Agent Vault reports (`14322`). Hosted ctxpipe on AWS sets it to the DNS name of the Agent Vault endpoint in the sandbox VPC, because AgentCore sandboxes cannot connect to the private name of the API. That endpoint must forward only the proxy port, never the API port (`14321`): code in a sandbox must not get access to the Agent Vault API. If you do not set it, sandboxes use the address of the `AGENT_VAULT_ADDR` host. Docker Compose and AWS CDK do not set it. | | `AGENTCORE_RUNTIME_ARN`, `AGENTCORE_RUNTIME_QUALIFIER` | Required when `SANDBOX_PROVIDER=agentcore`. Hosted ctxpipe uses this provider; AWS CDK does not support it yet. The ARN of the Amazon Bedrock AgentCore runtime that runs the `agentcore` target of the chat sandbox image, and its endpoint name (default `DEFAULT`). The backend also needs Agent Vault, `AGENT_VAULT_SANDBOX_PROXY_HOST` when the sandboxes are in a different VPC, and `SANDBOX_CALLBACK_DNS_SUFFIX` or `SANDBOX_CALLBACK_HOST`. Each process in the sandbox can read the credentials of the runtime's execution role at `169.254.169.254`. Thus that role must allow only the pull from the sandbox image's repository. It must also allow log group creation and log writes, only on the runtime's own log group. It must refuse each request that does not come through the sandbox VPC endpoints. AgentCore's own `BedrockAgentCore-*` sessions are the exception: AgentCore creates the log group from its own network. The ingest check allows that address because of this rule. | ### Agent Vault rules on AgentCore With `SANDBOX_PROVIDER=agentcore`, the Agent Vault network allowlist (`AGENT_VAULT_NETWORK_ALLOWLIST`) holds the backend's own subnets. Thus a sandbox can send requests to the backend task that runs its turn: to the model proxy on `PORT`, and to the tool bridge on a port that the kernel selects. Agent Vault has one allowlist for all its rules. It resolves the host of a rule again each time it connects. Thus the DNS of a host that a customer names, for example a plain git repository, can point at a backend task when Agent Vault connects. This is also possible when the host resolved to a public address at the time when the backend wrote the rule. The backend uses the port to keep these rules away from the backend's ports: * Only a callback rule can use a port other than 443. A callback rule is a model proxy or tool bridge rule that the backend makes, and its host is the backend's own callback name: `SANDBOX_CALLBACK_HOST`, or `ip----.` for an address of the backend. Each other rule gets port 443. * The backend refuses each other rule that names another port, for example `PORT`, a tool bridge port, or 80 (plain HTTP). * Agent Vault forwards a request only when a rule matches its host and its port, and it connects to that port. A DNS answer cannot change the port. * Each run vault refuses each host that no rule names. * When the backend writes a rule, it refuses a host that resolves to an address that is not public. If the host does not resolve, the sandbox does not start. This check is only a first layer, because the DNS answer can change after the check. The limit: a host that a customer names can still connect to port 443 of each address in the allowlist. Do not run a service that listens on port 443 in the backend's subnets, and do not add other subnets to the allowlist. Agent Vault 0.40.2 has no network allowlist for each rule, and no way to pin a rule to the addresses that the backend checked. With the other values of `SANDBOX_PROVIDER`, the backend writes the rules as they are. ### Plain git repositories A plain git repository is a repository that ctxpipe reads with no credential (see [Linked repositories](/docs/workspaces/linked-repositories#plain-git-repositories)). With `SANDBOX_PROVIDER=docker`, its URL can use `https://` or `http://`, another port, an IP address, or an internal host name. ssh (`git@…` or `ssh://`), `git://`, and `file://` URLs are not supported. ctxpipe never reads a host that resolves to a cloud metadata or link-local address (`169.254.0.0/16`, `fe80::/10`, or `fd00:ec2::254`). With each other value of `SANDBOX_PROVIDER`, or with no value, the URL must use `https://` on the default port, and each address of its host must be public. The backend and codesearch read the repository through the proxy of their environment (`HTTPS_PROXY`, `NO_PROXY`), if you set one. | Variable | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AGENT_VAULT_NETWORK_ALLOWLIST` | Docker Compose only (`.env`). Optional. The private addresses and CIDRs that Agent Vault may connect to, separated by commas. Compose always allows `172.30.99.10`, the backend, and adds your entries after it. To ingest a plain git repository on a private git server, add only the single host address of your git server, for example `10.20.0.15`. Without it, the ingestion sandbox cannot fetch the repository. Chat-sandbox vaults forward each host that no rule matches. An address or CIDR in this list therefore lets every chat agent reach every port and path at that address. Add only the single host address of the git server, never a whole subnet. Never add a range that covers `172.30.99.0/24`, and never add a link-local (`169.254.0.0/16`), loopback, or cloud metadata range. On AWS CDK, use `sandboxHost.agentVaultNetworkAllowlist` (see [AWS](/docs/self-hosting/deployment/aws#chat-sandbox-host)). | ## GitHub App connector Repository connectors use a GitHub App that your deployment owns. The in-product wizard stores the ID, the PEM private key, and the webhook secret of the App, encrypted, for the connection. | Variable | Description | | ----------------------------------- | --------------------------------------------------------------------------------- | | `GITHUB_APP_ID` | Optional after migrations. GitHub App numeric ID. | | `GITHUB_PRIVATE_KEY` | Optional after migrations. PEM private key for the App. | | `GITHUB_WEBHOOK_SECRET` | Optional after migrations. Also used for the legacy webhook route. | | `GITHUB_APP_SLUG` | Optional. Public app slug used for default install URLs. | | `CONNECTION_SECRETS_ENCRYPTION_KEY` | Optional 64-character hex key. When unset, encryption derives from `AUTH_SECRET`. | The wizard reserves the connection-specific webhook URL and gives the exact value to paste into GitHub. `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` are for signing in with GitHub and for the one-time authorization that proves access to an installation. Use the client ID and secret of the App in `GITHUB_APP_ID`. GitHub App credentials are for repository installation and webhook access. See [GitHub](/docs/self-hosting/github) for App registration, permissions, events, credentials, installation, and end-to-end verification. ## Atlassian and Confluence Self-hosted Confluence uses the same product wizard as hosted ctxpipe. But your deployment or organization must first provide the Atlassian and Forge settings. | Variable | Description | | ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | `ATLASSIAN_CLIENT_ID`, `ATLASSIAN_CLIENT_SECRET` | Optional global Atlassian 3LO app for account linking. If unset, each Forge connection can store its own 3LO credentials through the wizard. | | `CONFLUENCE_FORGE_INSTALL_URL` | Optional deployment-wide Forge install URL fallback when a Forge connection has no install URL in `connections.config`. | See [Confluence & Atlassian](/docs/self-hosting/confluence-atlassian) for the full operator flow. ## Slack Self-hosted Slack uses one Slack app that your deployment owns. Organizations authorize that app. They do not create per-tenant Slack apps or Event Subscription URLs. | Variable | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `SLACK_CLIENT_ID`, `SLACK_CLIENT_SECRET` | Slack app OAuth credentials. Required to enable the Slack connector. | | `SLACK_SIGNING_SECRET` | Verifies Event Subscription requests (`X-Slack-Signature`). | | `SLACK_REDIRECT_URI` | Optional. Override when the public OAuth callback differs from `{AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback`. | Set the client ID, client secret, and signing secret together. Then restart or redeploy the backend. The backend and the worker must share the same `AUTH_SECRET` and, when configured, `CONNECTION_SECRETS_ENCRYPTION_KEY`. Then the background capture jobs can use the encrypted Slack bot tokens. Set the Slack Event Subscriptions Request URL to: ```text https:///api/v1/webhook/slack ``` See [Slack](/docs/self-hosting/slack) for scopes, `app_mention` Events, and intent-based thread capture. ## Linear ctxpipe can store the Linear OAuth application and the webhook signing secret on the connection (Register in the product UI). `LINEAR_*` is an optional deployment-wide fallback. | Variable | Description | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | | `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET` | Optional shared Linear OAuth application credentials. When both are set, the wizard skips Register. | | `LINEAR_REDIRECT_URI` | Optional exact callback override. Defaults to `{AUTH_BASE_URL}/api/v1/integrations/linear/callback`. | | `LINEAR_WEBHOOK_SECRET` | Optional shared signing secret for `/api/v1/webhook/linear`. Per-connection secrets are stored on the row. | See [Linear](/docs/self-hosting/linear) for application registration, webhook subscriptions, preview limitations, and troubleshooting. ## Notion Notion uses one public integration and one webhook subscription. Your deployment owns both. | Variable | Description | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------- | | `NOTION_CLIENT_ID`, `NOTION_CLIENT_SECRET` | Public integration OAuth credentials. Both are required to enable the connector. | | `NOTION_REDIRECT_URI` | Optional exact callback override. Defaults to `{AUTH_BASE_URL}/api/v1/connectors/notion/oauth/callback`. | | `NOTION_WEBHOOK_SECRET` | Verification token shown by Notion after the first webhook delivery. Required for signed updates. | Put the OAuth credentials on every service that refreshes tokens or does connector work. Then register the single deployment webhook. See [Notion](/docs/self-hosting/notion) for provisioning and verification. ## PagerDuty PagerDuty can use an OAuth application that your deployment owns. After a user authorizes an account, ctxpipe creates the signed webhook subscription. These variables are an optional shortcut, as on hosted ctxpipe. A self-hosted deployment can instead save the app on the connection in the setup wizard. | Variable | Description | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | | `PAGERDUTY_CLIENT_ID`, `PAGERDUTY_CLIENT_SECRET` | Optional shared PagerDuty OAuth application. When unset, save client id and secret on the connection in the wizard. | | `PAGERDUTY_REDIRECT_URI` | Optional exact callback override. Defaults to `{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback`. | There is no `PAGERDUTY_WEBHOOK_SECRET`. ctxpipe stores the signing secret of each subscription encrypted on the connection. See [PagerDuty](/docs/self-hosting/pagerduty) for application registration and verification. ## Observability Telemetry is off until you set it. ctxpipe does not ship a collector or ClickStack in the Compose `deploy` profile or in `@ctxpipe/aws-cdk`. Unset values keep logs on stdout and do not start trace or metric export. Empty values count as unset. Point the signal URLs at your own OTLP/HTTP backend. Export starts from these variables. Include the signal path on each URL. ```bash OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com/v1/traces OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://otel.example.com/v1/logs OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://otel.example.com/v1/metrics OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer xxx ``` | Variable | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Traces URL, including `/v1/traces`. Trace export stays off when this is unset. | | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Logs URL, including `/v1/logs`. Unset keeps logs on stdout. | | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Metrics URL, including `/v1/metrics`. | | `OTEL_EXPORTER_OTLP_HEADERS` | Optional headers, for example `Authorization=Bearer xxx`. | | `OTEL_SERVICE_NAME` | Resource `service.name`. Compose and CDK set `backend`, `openworkflow`, `ui`, and `codesearch` per container. Compose does not apply a shared name from the root `.env`. | | `OTEL_RESOURCE_ATTRIBUTES` | Extra resource attributes, for example `deployment.environment=production`. | The `deployment.environment` value applies to backend traces, metrics, and logs, to codesearch, and to the UI relay. It is `DEPLOYMENT_ENVIRONMENT` when that is set. Hosted ctxpipe sets this variable; a self-hosted install leaves it unset, because a `pr-N` value turns on preview behavior (no email verification). Otherwise it is `deployment.environment` from `OTEL_RESOURCE_ATTRIBUTES`. Otherwise it is `production` when `NODE_ENV=production`. Otherwise it is `development`. Backend and codesearch also set `service.namespace=ctxpipe`. Codesearch always exports `service.name` `codesearch`. LLM spans are OpenTelemetry spans with the GenAI semantic conventions: `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, and `gen_ai.usage.cost` (USD, when the model gateway reports it). A span also holds the prompt and the output text in `gen_ai.input.messages` and `gen_ai.output.messages`. The backend logs never hold them. `@ctxpipe/aws-cdk` takes one base `otel.endpoint` and appends `/v1/traces`, `/v1/logs`, and `/v1/metrics`. See [AWS](/docs/self-hosting/deployment/aws). ### Browser telemetry The UI enables browser telemetry only when `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set on the UI server. The browser posts traces to the same origin at `/.otel/v1/traces`. The relay also accepts `/.otel/v1/logs`, but the SDK does not use it. Replay is off, and the SDK emits console capture as spans. The UI server forwards traces to `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and logs to `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, using `OTEL_EXPORTER_OTLP_HEADERS`. When the traces endpoint is unset, the browser SDK does not start. A signal whose endpoint is unset returns `204`. ## Advanced | Variable | Description | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `AUTH_ISSUER` | Override the OAuth issuer claim. Defaults to `AUTH_BASE_URL`. | | `AUTH_ALLOWED_ORIGINS` | Comma-separated list of allowed CORS origins for auth endpoints. | | `OPENWORKFLOW_CONCURRENCY` | In-flight OpenWorkflow runs per worker process. Default `4` when unset. Compose sets `6`. AWS CDK sets a value for each size. | | `AUTH_TOKEN_AUDIENCE_CODESEARCH` | Optional. The audience of the tokens that the backend sends to codesearch. If you set it, set the same value on the backend, the worker, and codesearch. | | `CODESEARCH_INDEXER_CONCURRENCY` | Max concurrent Zoekt/SCIP child processes on codesearch, and SCIP HTTP batch size on the worker. Default `2`. At most one of them runs a SCIP indexer at a time. | | `CODESEARCH_INDEX_PIPELINE_CONCURRENCY` | Max distinct repos with in-flight OpenWorkflow index-phase HTTP (clone included). Default `2`. Codesearch returns 429 when full. | ## Related pages # Graph databases (https://docs.ctxpipe.ai/docs/self-hosting/graph-databases) When you self-host this service, you must set up an OpenCypher-compatible graph database for knowledge graphs, RAG, and related workloads. This guide helps you choose and set up one of the supported engines. ## Quick Start (FalkorDB) The easiest way to start is **FalkorDB**. The default stack includes it as a Docker container. You do not need extra setup. Run: ```bash docker compose --profile deploy up -d ``` By default, the backend connects to FalkorDB with the **native driver** at `redis://falkordb:6379`. For host-based deployments, point `GRAPH_DB_URI` at your FalkorDB instance (for example `redis://localhost:6379`). Or use any OpenCypher-compatible database with the Bolt driver. On Docker Compose, the `GRAPH_DB_*` values in `.env` do not reach the backend. Set them in a `docker-compose.override.yml`. See [Settings that need an override file](/docs/self-hosting/deployment/docker#settings-that-need-an-override-file). ## Supported Engines You can use any of these OpenCypher-compatible engines. FalkorDB uses its native driver (Redis protocol). Others use Bolt. | Engine | Best for | Notes | | -------------------- | ----------------------------- | ------------------------------------------------------------ | | **FalkorDB** | Easy start, multi-tenant SaaS | Open-source. Included in Docker Compose. Graphs auto-create. | | **Neo4j Enterprise** | Enterprise multi-tenant | Requires `CREATE DATABASE` per org (admin). | | **Neo4j Community** | Cost-conscious multi-tenant | One database per instance. Use instance-per-tenant. | | **Memgraph** | Enterprise, real-time | Enterprise edition for multi-database. | | **Neptune** | AWS-hosted | Managed service. Single graph per cluster. | Set `GRAPH_DB_PROVIDER` to match your engine. The service infers the tenancy mode automatically. ## Environment Variables | Variable | Required | Default | Description | | ------------------------ | ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------- | | `GRAPH_DB_URI` | No | `redis://falkordb:6379` | FalkorDB: `redis://host:6379`. Others: Bolt URI (e.g. `bolt://host:7687`). | | `GRAPH_DB_USER` | No | `default` / `neo4j` | Username for auth. | | `GRAPH_DB_PASSWORD` | No | `""` | Password for auth. | | `GRAPH_DB_PROVIDER` | No | `falkordb` | Engine: `falkordb`, `neo4j-enterprise`, `neo4j-community`, `memgraph`, `neptune`. | | `GRAPH_DB_URI_` | Yes (instance-per-tenant) | - | Per-org URI. Required when using Neo4j Community or Neptune. No fallback to `GRAPH_DB_URI`. | **Example (FalkorDB, Docker network):** ```bash GRAPH_DB_URI=redis://falkordb:6379 GRAPH_DB_PROVIDER=falkordb ``` **Example (Neo4j Community, instance-per-tenant, org slugs `acme`, `piedpiper`):** ```bash GRAPH_DB_URI=bolt://neo4j.example.com:7687 GRAPH_DB_URI_acme=bolt://neo4j-acme.example.com:7687 GRAPH_DB_URI_piedpiper=bolt://neo4j-piedpiper.example.com:7687 GRAPH_DB_USER=neo4j GRAPH_DB_PASSWORD=yourpassword GRAPH_DB_PROVIDER=neo4j-community ``` **Example (Neptune):** ```bash GRAPH_DB_URI_acme=bolt+s://your-cluster.region.neptune.amazonaws.com:8182 GRAPH_DB_PROVIDER=neptune ``` ## Tenancy Modes All setups are **multi-tenant**. The service must isolate the graph data of each organization from the data of the others. The service infers the tenancy strategy from the provider in `GRAPH_DB_PROVIDER`. ### Database-per-tenant (shared instance) **Providers**: FalkorDB, Neo4j Enterprise, Memgraph These engines support **multiple graphs or databases in one instance**, with strong isolation between them. One shared instance serves many organizations. Each organization gets its own graph or database (for example `org_abc123`). The service passes the database name of the organization when it runs queries. Data therefore stays isolated at the database layer. ### Instance-per-tenant (separate instance per org) **Providers**: Neo4j Community, Neptune Neo4j Community and Neptune **do not support multiple databases or graphs in one instance**. Each instance has one logical graph. For per-organization isolation, the service uses **instance-per-tenant**. Each organization must have its own database instance. The service routes each query to the instance of the organization. The deployment topology enforces isolation, not the database engine. Set `GRAPH_DB_URI_` for each organization (for example `GRAPH_DB_URI_acme`, `GRAPH_DB_URI_piedpiper`). There is no fallback to `GRAPH_DB_URI`. Each organization must have its own URI. The AWS CDK construct sets this variable for its one `orgSlug`, so CDK users need to do nothing. ## Engine-Specific Setup ### FalkorDB * **Docker**: `falkordb/falkordb`. Uses Redis protocol on port 6379 by default. Pin a version in production; hosted ctxpipe pins one. * **URI**: `redis://host:6379` (native driver, recommended for production). * **User**: `default` * **Password**: Empty by default. With Redis `--requirepass`, use that password. ### Neo4j * **URI**: `bolt://host:7687` * **User**: `neo4j` * **Password**: Set via `GRAPH_DB_PASSWORD` or `NEO4J_AUTH` (Docker). ### Memgraph * **URI**: `bolt://host:7687` * **User**: Typically `memgraph` or as per your deployment. * **Enterprise**: Use `memgraph` provider for database-per-tenant. ### Neptune * **URI**: The Bolt endpoint of your cluster, with TLS (e.g. `bolt+s://cluster.region.neptune.amazonaws.com:8182`). * **Auth**: ctxpipe does not support IAM database authentication. Turn IAM auth off on the cluster, as the AWS CDK construct does. Set `GRAPH_DB_USER` and `GRAPH_DB_PASSWORD` only if your cluster needs them. See [AWS Neptune docs](https://docs.aws.amazon.com/neptune/). # MCP (https://docs.ctxpipe.ai/docs/self-hosting/mcp) Self-hosted ctxpipe exposes the same organization-scoped Streamable HTTP endpoint as hosted ctxpipe: ```text {AUTH_BASE_URL}/mcp ``` Every client uses this one URL. During authorization, the user chooses the organizations that the client can read. Each tool call names one organization in the `org` argument. The server accepts the old `?orgSlug=` parameter and ignores it. `AUTH_BASE_URL` must be the public origin clients use, including TLS. Client JSON shapes and OAuth install paths are the same as on hosted ctxpipe. See [Connect your agent](/docs/mcp/mcp-docs) and [Authentication](/docs/self-hosting/authentication). This page is for operators. Confirm that the endpoint is reachable. Then debug OAuth and tools on a **test organization**. `npx ctxpipe doctor mcp` checks HTTP routing, TLS or reachability, the unauthenticated Bearer challenge, and OAuth discovery. A `ready-for-oauth` result does **not** prove that browser OAuth, authenticated `tools/list`, or an MCP tool call work. The command does not accept OAuth bearer tokens and does not test STDIO servers. When `CTXPIPE_API_KEY` is set, it sends the key as `x-api-key` and checks an authenticated initialize. ## Connect a client Use the same config as [Quickstart](/docs/self-hosting/quickstart), with your deployment origin: ```json { "mcpServers": { "ctxpipe": { "type": "http", "url": "https://ctx.example.com/mcp" } } } ``` The MCP client must complete OAuth against this deployment. CLI setup sign-in and MCP client OAuth are separate flows. ## Diagnose discovery Run the lightweight doctor against the public MCP URL: ```bash npx ctxpipe doctor mcp \ --url "$AUTH_BASE_URL/mcp" npx ctxpipe doctor mcp \ --url "$AUTH_BASE_URL/mcp" \ --json ``` `--json` is machine-readable. Failures set a non-zero exit code. Loopback HTTP works with a warning. Remote endpoints must use HTTPS. | Check | Passes when | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | Target | Path is `/mcp`; remote origins use HTTPS | | Backend status | `GET /.status` returns JSON with `status: "ok"` | | OAuth challenge | Unauthenticated Streamable HTTP `initialize` returns `401` with a Bearer `WWW-Authenticate` challenge that includes `resource_metadata` | | Protected resource | RFC 9728 metadata matches the `/mcp` audience and names an authorization server | | Authorization server | RFC 8414 metadata exposes authorization, token, and registration endpoints | Overall status is `ready-for-oauth`, `ready-for-api-key` (with `CTXPIPE_API_KEY` set), `warning`, or `failed`. If TLS fails from a Node CLI against a local CA (Portless or a private CA), set `NODE_EXTRA_CA_CERTS` to the CA file. Do not disable TLS verification. ## Authenticated checks with MCPJam MCPJam is an external diagnostic client. It is not bundled into `ctxpipe` or the self-hosted backend. Pin the versions below. Do not use `@latest`. Use a test organization. Do not use a customer organization. ```bash npx @mcpjam/cli@3.24.0 oauth login \ --url "$AUTH_BASE_URL/mcp" \ --protocol-version 2025-11-25 \ --registration dcr \ --credentials-out .mcpjam/credentials.json \ --debug-out .mcpjam/oauth-debug.json npx @mcpjam/cli@3.24.0 tools list \ --url "$AUTH_BASE_URL/mcp" \ --credentials-file .mcpjam/credentials.json npx @mcpjam/cli@3.24.0 tools call \ --url "$AUTH_BASE_URL/mcp" \ --credentials-file .mcpjam/credentials.json \ --tool-name list_workspaces \ --tool-args '{}' ``` For a visual OAuth and JSON-RPC timeline on a local origin: ```bash npx @mcpjam/inspector@2.40.0 \ --url "$AUTH_BASE_URL/mcp" \ --name "ctxpipe self-host" \ --oauth \ --tab servers ``` The hosted MCPJam application only reaches public HTTPS. It cannot use `localhost` or refresh against a local authorization server. ## Artifacts and credentials Keep OAuth credentials and debug traces out of Git. Write them under `.mcpjam/` (gitignored in the ctxpipe repository). Review files before sharing. The credentials file can include an access token, refresh token, client id, and client secret even when stdout is redacted. The ctxpipe doctor never accepts or prints OAuth bearer tokens, and never prints an API key. Do not put `access_token` on the MCP URL. MCPJam does not prove token refresh races, dropped-session recovery, or concurrent clients. Do not gate CI on interactive OAuth or LLM evals. ## Common failures * **`org_not_found` or `org_required`** — the tool call named an organization that is not in the connection, an organization that the user is not a member of, or no organization. Pass the right slug in `org`. For an OAuth connection, add the organization on **Connected apps**, or reconnect the client and select the organization during authorization. * **`AUTH_BASE_URL` mismatch** — issuer, callbacks, and MCP discovery must use the same public origin clients see. * **Remote HTTP** — production MCP must be HTTPS. * **Status unreachable or timed out** — backend, DNS, TLS, or an untrusted CA. * **HTTP 401 without `resource_metadata`** — clients cannot start OAuth discovery. * **Protected-resource audience mismatch** — RFC 9728 `resource` canonicalizes to `/mcp` without the query string. * **Authorization-server metadata missing registration** — dynamic client registration cannot proceed. * **CLI setup sign-in worked, but the MCP client still asks for authorization** — expected. Setup auth and MCP OAuth are separate. ## Related pages # Model configuration (https://docs.ctxpipe.ai/docs/self-hosting/models) When you self-host ctxpipe, you set the LLM and embedding models with environment variables. The service supports any OpenAI-compatible HTTP API. Use **`MODEL_PROVIDER`** to select OpenRouter-specific features, Azure (`api-key` auth), or Bedrock (native SDK with IAM SigV4). ## `MODEL_PROVIDER` values | Value | Auth (chat + embeddings) | `MODEL_PROVIDER_URL` default | Notes | | ------------- | ------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `openai-like` | `Authorization: Bearer` | `https://openrouter.ai/api/v1` | Generic compatible API; no OpenRouter-only extras. | | `openrouter` | `Authorization: Bearer` | same | Enables plugins, prompt cache, reasoning flags, medium-tier model fallbacks. | | `azure` | `api-key` header only | **required** | Set your Azure OpenAI OpenAI-compatible base URL; override model name env vars for deployment IDs. | | `bedrock` | IAM SigV4 (ECS task role) | not required | Native Bedrock Runtime via AWS SDK. Set `MODEL_BEDROCK_AWS_REGION` or rely on `AWS_REGION` / `AWS_DEFAULT_REGION`. No `MODEL_PROVIDER_API_KEY` on AWS CDK deployments. | If you do not set **`MODEL_PROVIDER`**, it defaults to **`openai-like`**. To keep the **previous** behavior (OpenRouter URL heuristics that added extras whenever the host was OpenRouter), set **`MODEL_PROVIDER=openrouter`**. For Vertex and other gateways that expect Bearer and a base URL, use **`openai-like`**. If you route through OpenRouter, use **`openrouter`**. When you deploy with [`@ctxpipe/aws-cdk`](/docs/self-hosting/deployment/aws) and `modelProvider.kind: "bedrock"`, the construct does **not** store a static `MODEL_PROVIDER_API_KEY`. Backend and worker tasks receive IAM permissions (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`) and call Bedrock Runtime through the AWS SDK. They use the ECS task role (SigV4). Enable your chosen model IDs in the Bedrock console for the stack region before you use them. If the account is in an AWS Organization, also confirm that **service control policies (SCPs)** allow those same foundation-model ARNs. An SCP explicit deny overrides the task role. It shows as `AccessDeniedException` … `explicit deny in a service control policy`. When `MODEL_PROVIDER=bedrock`, embeddings use the native Bedrock Runtime API. Only **Cohere** embed models work. Other Bedrock embedding models do not work yet. On AWS CDK, the default is `cohere.embed-v4:0` when you do not set `models.embedding`. You can also use an inference-profile ID, for example `global.cohere.embed-v4:0`. If an SCP denies a profile ID, the error often names the base `…/foundation-model/cohere.embed-v4:0` ARN. ## Quick Start (OpenRouter) The default setup uses OpenRouter. Set your API key and start the stack: ```bash MODEL_PROVIDER_API_KEY=sk-or-v1-... docker compose --profile deploy up -d ``` LLM and embeddings both use OpenRouter by default. You do not need extra configuration. ## Environment Variables | Variable | Required | Default | Description | | -------------------------- | -------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `MODEL_PROVIDER` | No | `openai-like` | `openai-like` \| `openrouter` \| `azure` \| `bedrock` - see table above. | | `MODEL_PROVIDER_API_KEY` | Yes\* | - | API key for OpenAI-compatible providers. Not used for native Bedrock (`MODEL_PROVIDER=bedrock`). | | `MODEL_PROVIDER_URL` | No\*\* | OpenRouter base when not `azure` | Base URL for OpenAI-compatible LLM/embeddings HTTP. **Required** for `azure`. Not required for native Bedrock. | | `MODEL_BEDROCK_AWS_REGION` | No | `AWS_REGION` / stack region | AWS region for Bedrock Runtime when `MODEL_PROVIDER=bedrock`. | | `MODEL_PROVIDER_ALLOWLIST` | No | - | OpenRouter provider slugs, separated by commas (for example `openai,anthropic,azure`). When set, each OpenRouter call goes only to these providers. When unset, OpenRouter can use any provider. It needs `MODEL_PROVIDER=openrouter` or a `MODEL_PROVIDER_URL` on `openrouter.ai`. A value that is not a slug list, or other model settings, stop the backend at start. | | `MODEL_FAST_NAME` | No | `openai/gpt-6-luna?reasoning.effort=high` | LLM model for the fast tier (OpenRouter slash ids). | | `MODEL_MEDIUM_NAME` | No | `openai/gpt-6-luna?reasoning.effort=xhigh` | LLM model for the medium tier. | | `MODEL_HIGH_NAME` | No | `openai/gpt-6-luna?reasoning.effort=xhigh` | LLM model for the high tier. | | `MODEL_EMBEDDING_NAME` | No | `openai/text-embedding-3-large` | Embedding model ID. | | `MODEL_VISION_NAME` | No | `openai/gpt-6.1-terra` | The one OpenRouter model that reads the images of the knowledge plane. It must take image input and have a ZDR endpoint on OpenRouter. A value of only spaces stops the backend at start. | \*Required for each provider except `bedrock`. The backend refuses model calls without it. For a provider that needs no key, such as Ollama, set any value that is not empty. \*\*Default `MODEL_PROVIDER_URL` is `https://openrouter.ai/api/v1` for `openai-like` and `openrouter` so bundled default model IDs stay valid. Each model call to OpenRouter allows only zero data retention (ZDR) endpoints, with no data collection. A model with no ZDR endpoint fails. A call goes to OpenRouter when `MODEL_PROVIDER` is `openrouter`, or when `MODEL_PROVIDER_URL` is on `openrouter.ai`. Another OpenAI-compatible API does not get these routing fields. ## How to Pick Models ### LLM tiers The service uses three tiers for different workloads: | Tier | Use case | Default model spec | | ---------- | --------------------------------- | ------------------------------------------ | | **fast** | Quick tasks, naming, planning | `openai/gpt-6-luna?reasoning.effort=high` | | **medium** | Main agent, balanced cost/quality | `openai/gpt-6-luna?reasoning.effort=xhigh` | | **high** | Complex reasoning, best quality | `openai/gpt-6-luna?reasoning.effort=xhigh` | Override any tier with `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, or `MODEL_HIGH_NAME`. Use model IDs from your provider (OpenRouter: `org/model-name`; Bedrock on AWS CDK: `vendor.model` with dot, e.g. `openai.gpt-6-luna?reasoning.effort=high`). ### Model spec query params Append **`?reasoning.effort=value`** to set the reasoning depth of a tier without extra environment variables. The supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Each provider adapter maps this value to its upstream API (`reasoning_effort` on OpenAI-like / Azure / Bedrock Chat Completions; nested `reasoning.effort` on OpenRouter). ```text openai/gpt-6-luna?reasoning.effort=high openai.gpt-6-luna?reasoning.effort=xhigh ``` To disable reasoning in environment specs, use **`reasoning.effort=none`**. Do not use `reasoning=false`. Call sites can also pass `getModel(..., { reasoning: false })`, which merges `reasoning.effort=none` over the tier spec. ctxpipe sends model ids upstream without query strings. When **`MODEL_PROVIDER=openrouter`** (or hosted ctxpipe, which sets it), the **medium** tier sends the OpenRouter `models` array. If the primary model fails (rate limits, downtime, moderation), routing tries the next distinct model IDs in order (**`MODEL_FAST_NAME`**, then **`MODEL_HIGH_NAME`**). ctxpipe removes duplicate base model ids from that chain. This does **not** apply to `openai-like`. ### Embedding model **Important:** The embedding model must support **2000 dimensions**. The schema stores vectors in `vector(2000)`; a model that is not compatible fails. * **OpenAI-compatible providers** (OpenRouter, OpenAI, Vertex): Use models that accept a `dimensions` parameter, for example `openai/text-embedding-3-large` (3072 dims, truncated to 2000). * **Amazon Bedrock**: Use a **Cohere** embed model ID (for example `cohere.embed-v4:0` or `global.cohere.embed-v4:0`). Other Bedrock embed families are not supported yet. * **Ollama**: Use models with native 2000-dim support, e.g. `qwen3-embedding`. Set `MODEL_PROVIDER_URL=http://host:11434/v1` and `MODEL_EMBEDDING_NAME=qwen3-embedding` (embeddings use `{MODEL_PROVIDER_URL}/embeddings`). All providers use the same OpenAI-compatible embeddings API. Ollama exposes it at `/v1/embeddings`. ## Provider Examples ### OpenRouter (default host) ```bash MODEL_PROVIDER_API_KEY=sk-or-v1-... # Optional explicit mode (enables OpenRouter extras; default URL is OpenRouter either way): # MODEL_PROVIDER=openrouter MODEL_PROVIDER_URL=https://openrouter.ai/api/v1 ``` ### Azure OpenAI ```bash MODEL_PROVIDER=azure MODEL_PROVIDER_API_KEY= MODEL_PROVIDER_URL=https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT # Set model env vars to your deployment names as needed. ``` ### Amazon Bedrock (native SDK) ```bash MODEL_PROVIDER=bedrock MODEL_BEDROCK_AWS_REGION=us-east-1 MODEL_EMBEDDING_NAME=cohere.embed-v4:0 # or global.cohere.embed-v4:0; Cohere only on Bedrock # ECS task role / instance profile supplies AWS credentials; no MODEL_PROVIDER_URL or API key. # Org SCPs must allow the underlying foundation-model ARN (see AWS CDK Bedrock callout above). ``` ### Ollama (local chat + embeddings) ```bash MODEL_PROVIDER_URL=http://localhost:11434/v1 MODEL_PROVIDER_API_KEY=ollama # Ollama needs no key, but the value must not be empty MODEL_FAST_NAME= MODEL_MEDIUM_NAME= MODEL_HIGH_NAME= MODEL_EMBEDDING_NAME=qwen3-embedding # For Docker: use http://host.docker.internal:11434/v1 ``` Set each model name. The defaults are OpenRouter model ids, which Ollama does not know. ## Resources * [OpenRouter models](https://openrouter.ai/models) * [OpenRouter embeddings](https://openrouter.ai/docs/api/reference/embeddings) * [Ollama OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility#%2Fv1%2Fembeddings) # Architecture (https://docs.ctxpipe.ai/docs/self-hosting/architecture) Self-hosted ctxpipe uses the same product architecture as hosted ctxpipe. But the operator owns the runtime, network boundary, secrets, and stateful services. Keep the browser, REST API, auth callbacks, and MCP endpoint on the same public backend origin. ## Request flow 1. The browser opens your public `AUTH_BASE_URL`. 2. The backend serves API, auth, webhook, and MCP routes from that origin. 3. The backend proxies UI requests to the `ui` service. 4. Auth sessions, repository records, connections, and workflow state live in Postgres. 5. Repository ingestion calls `codesearch`, which clones repositories and maintains Zoekt indexes. Codesearch is a **single replica**. Zoekt shards, git clone cache, and SCIP artifacts live on one process and one volume. Extra worker replicas do not raise ingest throughput unless codesearch memory (and `OPENWORKFLOW_CONCURRENCY` / indexer caps) also grow. 6. The worker runs durable jobs for ingestion, connector setup, scoped source mirrors, Slack thread capture, and Forge provisioning. 7. Extracted claims and relationships are projected into the configured graph database. For an organization on the new knowledge store, the worker also writes the documents, their sections, and their relations to ClickHouse. 8. Chat and MCP tools retrieve from Postgres, codesearch, and the graph database (or ClickHouse, for an organization on the new knowledge store) before calling your configured model provider. ## Services | Service | Default port | Stateful | Purpose | | ----------------------------- | ---------------------------------------------------: | -------- | ------------------------------------------------------------ | | `backend` | `3000` | No | Public API, auth, MCP, callbacks, UI proxy | | `ui` | `3002` | No | Product web app, proxied by backend | | `worker` | internal | No | Background workflows and connector jobs | | `codesearch` | `3001` | Yes | Repository clone cache and Zoekt index. Stay at one replica. | | `postgres` | `5432` internal (no host port in the deploy profile) | Yes | Primary relational state | | `falkordb` | `6379` Redis, `7687` Bolt, internal | Yes | Default graph database | | `clickhouse` (Docker Compose) | `8123` HTTP, internal | Yes | The new knowledge store | | `dind` (Docker Compose) | `2376` internal | No | Docker daemon for Workspace chat sandboxes | ### Chat sandbox network policy The same policy holds on Docker Compose and AWS: * A chat sandbox holds no credential. It has no GitHub token, no model proxy credential, and no tool bridge token in its environment, files, or Git configuration. [Agent Vault](https://github.com/Infisical/agent-vault) (open source) runs next to the sandboxes and adds each credential to the requests that need it. * Sandboxes reach the network only through the Agent Vault proxy. On Compose, rules in the `dind` service enforce this. On AWS, `DOCKER-USER` rules on the sandbox host enforce it. HTTP and HTTPS stay open. Agents can clone, fetch, push, call the GitHub CLI, install packages, and read documentation on the internet. The proxy forwards hosts that have no credential rule without change. * The proxy blocks all other traffic from a sandbox. This includes a direct connection that does not use the proxy, any non-HTTP protocol (SSH, raw TCP, UDP), and DNS. Sandboxes do not resolve names. The proxy resolves them. * The proxy allows private addresses as follows. On Compose, it allows only the fixed address of the backend on the `sandbox` network. On AWS, it allows only the subnets of the backend. The host also rejects traffic from Agent Vault to the host itself. Agent Vault never reaches its own management API. Sandboxes never reach data stores. Compose never publishes Postgres or FalkorDB. AWS admits only the security group of the app to them. * Sandboxes cannot reach cloud instance metadata (`169.254.169.254`). * Each agent port needs a password, and sandboxes cannot reach each other over the Docker bridge. * Each chat turn gets its own Agent Vault vault and proxy session. The backend deletes the vault when the turn ends, and the session stops with it. The backend also revokes the turn's GitHub token. * If Agent Vault does not answer, Docker chat fails closed with a clear error. The backend never gives a sandbox a credential instead. The deployment generates the Agent Vault passwords. You do not set them. On Compose, the master password is on a volume that only Agent Vault mounts, and the owner password is on a volume that only the backend and the worker mount. On AWS, the stack keeps both in Secrets Manager. Only the host reads the master password. #### Agent Vault owner password The backend registers the owner password on its first login to an Agent Vault with no owner. After that, Agent Vault refuses another password. Docker chat then fails closed with an error that names this section. To change the password, first change it in Agent Vault (`agent-vault account change-password`, or its web UI on port 14321 inside the deployment). Then give the backend the new value. To start over, remove Agent Vault's data (Compose: the `agent_vault_data` volume; AWS: replace the sandbox host). Run vaults are temporary, so no data is lost. Tools in a sandbox must use the standard proxy variables (`HTTPS_PROXY`, `HTTP_PROXY`) and trust the proxy's certificate authority. The sandbox environment sets both. A tool that ignores the proxy variables cannot reach the network. ### Chat sandboxes on Docker Compose Compose runs each Workspace conversation's agent in its own container inside `dind`, a stock Docker-in-Docker service. `dind`, `agent-vault`, `backend`, and `worker` share a separate `sandbox` network (`172.30.99.0/24`): | Component | Port | Reached by | Purpose | | ----------------------------------------------- | -------------------------------------------------------------: | --------------------------------------- | ------------------------------------------------- | | Docker API (mutual TLS) | `2376` | `backend`, `worker` | Create, stop, resume, and remove sandboxes | | Sandbox agent ports (password per conversation) | `32768`–`60999` on `dind` | `backend` | The backend talks to the agent in each sandbox | | Agent Vault API | `14321` | `backend`, `worker` | One vault per chat turn | | Agent Vault proxy | `14322` | Sandboxes (the only address they reach) | Adds credentials and forwards HTTP(S) | | Backend model proxy and tool bridges | `3000` and per-run ports on `backend.sandbox.ctxpipe.internal` | Agent Vault, for sandboxes | Model calls and tool calls go back to the backend | See [Docker: Chat sandboxes](/docs/self-hosting/deployment/docker#chat-sandboxes). ### Chat sandbox host on AWS On AWS, `@ctxpipe/aws-cdk` adds one EC2 Docker host next to the Fargate services. Each Workspace conversation's agent runs in its own container there. | Component | Port | Reached by | Purpose | | ----------------------------------------------- | ----------------------: | --------------------------------------- | ---------------------------------------------------------------------------------------------- | | Docker API (mutual TLS) | `2376` | `backend`, `worker` | Create, stop, resume, and remove sandboxes | | Sandbox agent ports (password per conversation) | `32768`–`60999` | `backend` | The backend talks to the agent in each sandbox | | Agent Vault API | `14321` | `backend`, `worker` | One vault per chat turn | | Agent Vault proxy | `14322` (on the host) | Sandboxes (the only address they reach) | Adds credentials and forwards HTTP(S) | | Backend model proxy and tool bridges | `3000`, `32768`–`60999` | Agent Vault, for sandboxes | Model calls and tool calls go back to the backend task that runs the turn, at its VPC DNS name | `DOCKER-USER` rules on the host let sandboxes reach only the Agent Vault proxy. Agent Vault reaches the internet through NAT and may dial private addresses only in the VPC. Security groups keep Postgres, Neptune, ClickHouse, EFS, codesearch, the UI, and the worker away from the host. The host also blocks sandboxes from the host itself. See [AWS: Chat sandbox host](/docs/self-hosting/deployment/aws#chat-sandbox-host). ## Public surface Expose the backend origin publicly. Keep internal services private unless you have an operational need. | Route or endpoint | Public? | Notes | | ------------------------------------ | -------- | ---------------------------------------------------------------------------------------------- | | `/` | Yes | Backend proxies UI requests | | `/.auth/*` | Yes | Better Auth routes and provider callbacks | | `/api/v1/*` | Yes | REST API and connector callbacks | | `/:orgSlug/api/v1/*` | Yes | Organization-scoped REST API | | `/mcp` | Yes | Streamable HTTP MCP endpoint. Tool calls name the organization in `org` | | `/.status` | Optional | Basic backend status endpoint | | `ui:3002` | No | Internal service behind backend. Docker Compose binds it to `127.0.0.1` only. | | `codesearch:3001` | No | Internal service called by backend and workflows. Docker Compose binds it to `127.0.0.1` only. | | `postgres`, `falkordb`, `clickhouse` | No | Stateful infrastructure. Never published. | ## Stateful data Back up anything you cannot rebuild quickly. | Data | Location | Recovery note | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | App database | Postgres. On Compose, an anonymous volume unless you mount a named one (see [Docker](/docs/self-hosting/deployment/docker#persistent-state)). | Required. Contains users, sessions, repositories, connections, workflow state, and encrypted connector metadata. | | Graph data | FalkorDB or another graph database. On Compose, the writable layer of the `falkordb` container unless you mount a named volume (see [Docker](/docs/self-hosting/deployment/docker#persistent-state)). | Back up for fast recovery. Some graph data may be regenerated by reindexing, but that can be slow and may not cover all operational state. | | Knowledge content | ClickHouse. On Compose, the `clickhouse_data` volume. | Back up for fast recovery. ClickHouse holds only data that ctxpipe makes from the Workspace repositories, so a full rebuild of each Workspace can make it again. | | Repository cache | `repo_cache` volume in Compose, EFS `/data` on AWS CDK | Rebuildable by recloning, but useful to retain. | | Zoekt index | `zoekt_index_deploy` volume in Compose, EFS `/data` on AWS CDK | Rebuildable by reindexing, but can be expensive for large repositories. | | Chat sandboxes | `sandbox_docker` volume in Compose, Docker volume on the AWS sandbox host | No backup needed. Conversations keep their work in their git branch, and a lost sandbox is created again from it. | | Secrets | Deployment secret store or `.env` files | Required to decrypt sessions and connector credentials. Protect and version changes carefully. | ## Related pages # Quickstart (https://docs.ctxpipe.ai/docs/self-hosting/quickstart) This quickstart is the same for each deployment path. Deploy ctxpipe. Then test the product loop: sign in, connect GitHub, and index one repository. Then search it from an agent with the MCP tools. ## 1. Deploy For a Docker Compose evaluation, clone the repository and set the minimum values: ```bash git clone https://github.com/ctxpipe-ai/ctxpipe.git cd ctxpipe cp docker-compose.env.example .env # Set AUTH_SECRET, AUTH_BASE_URL, CTXPIPE_PUBLIC_APP_URL, # MODEL_PROVIDER, and MODEL_PROVIDER_API_KEY. pnpm start ``` `pnpm start` runs the Compose `deploy` profile. See [Docker](/docs/self-hosting/deployment/docker) for what it starts and what to set before people depend on it. For AWS, see [AWS CDK](/docs/self-hosting/deployment/aws). The browser, auth, the API, connector callbacks, and MCP all use the public backend origin, `AUTH_BASE_URL`. Keep the other services private. ## 2. Check the deployment Before you add a source, make sure that: * `GET {AUTH_BASE_URL}/.status` returns `ok`; * the app opens through `AUTH_BASE_URL`; * you can sign in with at least one method. See [Authentication](/docs/self-hosting/authentication); * the backend and the worker can reach the model provider; * Postgres and the graph database are healthy. ## 3. Create your GitHub App and a Workspace Open **Connectors**, select **Add connection**, then select **Connect** on the **GitHub** row. The wizard helps you create a GitHub App that your deployment owns. Follow [Self-host GitHub](/docs/self-hosting/github). Then install the App on a few repositories. Then [create a Workspace](/docs/workspaces/create-workspace), and [link one code repository](/docs/workspaces/linked-repositories) in its **Settings**. Wait until the repository shows **Indexed**. ## 4. Connect an agent Run the CLI with the URL of your deployment: ```bash npx ctxpipe init --base-url https://ctx.example.com ``` Or set up your client by hand with `https://ctx.example.com/mcp`. See [Connect your agent](/docs/mcp/mcp-docs). The client authorizes against your deployment. The signed-in user must be a member of each organization that the client reads. ## 5. Prove the loop Ask the agent: ```text Use ctxpipe before you plan. Call list_workspaces, then search_knowledge and search_code. Name the relevant architecture, standards, and source files, and say what still needs a check in the code. ``` You are ready to invite your team when: * `list_workspaces` lists your organization and its Workspaces; * `search_knowledge` and `search_code` return useful refs; * a repository that you index again finishes; * backups and logs are in place; * you have a rollback procedure for your deployment path. # Operations (https://docs.ctxpipe.ai/docs/self-hosting/operations) These docs assume that you already configured the deployment. Use this page to keep a running instance healthy and to diagnose component-level failures. ## Health checks | Check | Command or route | Expected result | | ------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- | | Backend status | `GET /.status` | JSON response with `status: "ok"` and `timestamp` | | Postgres in Compose | Docker health check uses `pg_isready` | Service is healthy before migrations and app services start | | UI through backend | Open `AUTH_BASE_URL` | App loads from the backend origin | | MCP route | `npx ctxpipe doctor mcp --url "$AUTH_BASE_URL/mcp"` then an OAuth-capable MCP client | Doctor reports `ready-for-oauth` (or `ready-for-api-key` when `CTXPIPE_API_KEY` is set); client OAuth then succeeds | | Codesearch | Add or reindex a repository | Repository reaches indexed state; backend can call `CODESEARCH_URL` | | Graph database | Open graph page or use graph-backed chat/MCP | Graph responses succeed; backend does not return graph connectivity errors | The reference codesearch service does not expose a separate public health endpoint. Use successful repository clone, index, and search operations as the health signal. ## Migrations The Compose `deploy` profile starts a `migrate` service before backend, worker, UI, and codesearch. Migrate uses the owner URL and provisions `ctxpipe_app`. App processes use the app-role `DATABASE_URL`. The migrate container runs: ```bash bun run apps/backend/src/db/migrate.ts ``` The AWS CDK migrate task and the hosted deploy run the same script, `migrate.ts`. It does these steps in this order. It makes the `ctxpipe_app` role ready. It applies the Drizzle and OpenWorkflow migrations. It runs every data backfill. Then it grants `ctxpipe_app` access to the tables. After the Drizzle migrations, migrate runs the backfills. The org backfill fills a missing `org_id` on rows that the previous release wrote (the `BEFORE INSERT` triggers on `repository_checkouts`, `confluence_spaces`, and `claim_evidence` do the same at write time). You do not run it by hand. See [Upgrades](/docs/self-hosting/upgrades#what-the-sql-changes) for the expand-only rules and the rollback runbook. Migrations are idempotent. Drizzle tracks applied migrations in the `drizzle.__drizzle_migrations` table and skips migrations that already ran. A second run changes nothing. Migrate reads the `ctxpipe_app` password from `DATABASE_APP_PASSWORD`: * When the password is set, migrate creates the role if it is missing and sets its password. * When the password is not set and the role exists, migrate only grants access. * When the password is not set and the role does not exist, migrate stops with exit code 1 before it changes the database. The error tells you how to fix it for each deployment path. Compose sends `POSTGRES_PASSWORD`. The CDK migrate task sends the app database credentials secret. For non-Compose deployments: 1. Back up Postgres first. 2. Run migrations once against the production database. 3. Start the new backend, worker, UI, and codesearch versions after migrations complete. ## Logs ctxpipe writes structured application logs through evlog. Without an OTLP logs endpoint, logs go to stdout. | Component | Useful signals | | ---------------- | -------------------------------------------------------------------------------------------------------------------------- | | `backend` | auth failures, REST errors, MCP requests, graph read failures, connector callbacks | | `worker` | ingestion jobs, Confluence, Linear, PagerDuty, Buildkite, and Plain sync, Forge provisioning, background workflow failures | | `codesearch` | clone failures, indexing failures, Zoekt errors | | `postgres` | connection limits, slow queries, disk pressure | | `graph database` | connection failures, memory pressure, query errors | | `clickhouse` | disk use of the hot volume, the count of pending mutations in `system.mutations`, query errors | Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` on the app services when you want telemetry to leave stdout and reach your observability backend. See [Configuration](/docs/self-hosting/configuration) for headers, service names, and LLM spans. ctxpipe does not run a collector for you. You see export failures in your collector or vendor. There is no bundled collector container. ## Backups Back up before upgrades, secret rotation, connector migrations, and storage changes. | Data | Backup guidance | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Postgres | Required. Use managed database backups or `pg_dump`/physical backups. Test restore regularly. | | Graph database | Back up or snapshot according to provider guidance. On Compose, mount a named volume first (see [Docker](/docs/self-hosting/deployment/docker#persistent-state)). | | ClickHouse | Optional. Back up `clickhouse_data` for fast recovery. Without a backup, a full rebuild makes the content again from the Workspace repositories (see [Recover the knowledge store](/docs/self-hosting/configuration#recover-the-knowledge-store)). | | `repo_cache` | Optional but recommended for large/private repositories. Rebuildable by recloning. | | `zoekt_index_deploy` | Optional but recommended for large deployments. Rebuildable by reindexing. | | Secrets | Export from your secret manager or platform. Keep historical values needed for rollback. | ## Chat sandbox host (AWS) `@ctxpipe/aws-cdk` runs Workspace chat sandboxes on one EC2 Docker host (see [AWS: Chat sandbox host](/docs/self-hosting/deployment/aws#chat-sandbox-host)). It holds no data that you need to back up. A replaced host starts empty, and conversations continue from their git branch. Find the instance from the `SandboxHostAutoScalingGroupName` stack output and open a shell with Session Manager (no SSH key or open port is needed): ```bash aws ssm start-session --target ``` | Check | Command on the host | Expected result | | ------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------ | | Docker is up | `sudo systemctl is-active docker` | `active` | | Docker volume | `df -h /var/lib/docker` | Below 80% used. The disk alarm fires at 80%. | | What uses the disk | `sudo docker system df` | Images, containers, and build cache sizes | | Sandboxes | `sudo docker ps -a` | One container per recent conversation; stopped ones are idle sandboxes | | Memory cap | `systemctl status ctxpipe-sandboxes.slice` | Memory in use stays below the slice limit (85% of host memory) | | Chat image | `sudo docker image ls ghcr.io/ctxpipe-ai/chat-sandbox` | The tag the backend's `SANDBOX_CHAT_IMAGE` names, after the first chat of a release | | Workspace bases | `sudo docker image ls ctxpipe-workspace-base` | At most about one image for each Workspace that had a new conversation in the last 30 days | | Boot log | `sudo tail -n 100 /var/log/cloud-init-output.log` | Ends with the CloudFormation success signal | Sandboxes stop after 5 minutes idle. ctxpipe removes them 30 days after last use. An organization runs at most 50 sandboxes at once. A Workspace base is an image of the Workspace repository after clone and setup. A new conversation then starts without a clone. The periodic sandbox cleanup deletes a base when a newer base replaces it. It deletes the current base when no new conversation started from it for 30 days. The same cleanup removes objects with ctxpipe labels (`ai.ctxpipe.*`) that no database row records. If the volume still fills, check `docker system df` to see what grows. Then raise `sandboxHost.dockerVolumeSizeGiB` and run `cdk deploy`. The new size replaces the host. Do not run `docker system prune` or `docker container prune` while conversations are active. These commands delete stopped sandboxes that conversations would resume. Do not run `docker image prune -a`. It deletes Workspace bases. The next new conversation of each Workspace then clones the repository again while a new base builds. ## Chat sandboxes (Docker Compose) The Compose deploy profile runs Workspace chat sandboxes inside the `dind` service (see [Docker: Chat sandboxes](/docs/self-hosting/deployment/docker#chat-sandboxes)). Its `sandbox_docker` volume holds no data that you need to back up. Run these commands from the repository root: | Check | Command | Expected result | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | Daemon is up | `docker compose --profile deploy ps dind` | `healthy` | | Chat image | `docker compose --profile deploy logs chat-sandbox-image` | Ends with the image ID (`sha256:…`) | | Sandbox disk | `docker compose exec dind df -h /var/lib/docker` | Room left on the host disk that holds the Docker volumes | | What uses the disk | `docker compose exec dind docker system df` | Images, containers, and build cache sizes | | Workspace bases | `docker compose exec dind docker image ls ctxpipe-workspace-base` | At most about one image for each Workspace that had a new conversation in the last 30 days | | Sandboxes | `docker compose exec dind docker ps -a` | One container per recent conversation; stopped ones are idle sandboxes | | Memory cap | `docker compose exec dind cat /sys/fs/cgroup/ctxpipe-sandboxes/memory.current /sys/fs/cgroup/ctxpipe-sandboxes/memory.max` | Memory in use stays below the cap. Missing on hosts without cgroup v2 (`dind` logs a warning). | | Metadata block | `docker compose exec dind iptables -t raw -S PREROUTING` | Includes `-d 169.254.169.254/32 -j DROP` | | No published data stores | `docker port $(docker compose ps -q postgres falkordb clickhouse)` | Prints nothing | | No host socket | `docker inspect --format '{{range .Mounts}}{{.Source}} {{end}}' $(docker compose ps -q backend worker)` | No `/var/run/docker.sock` | ctxpipe removes sandboxes and Workspace bases on the same schedule as on AWS. To free disk and keep the sandboxes, remove untagged chat images and build cache: ```bash docker compose exec dind docker image prune -f docker compose exec dind docker builder prune -f ``` `docker image prune` without `-a` keeps every tagged image, so it keeps the Workspace bases. Do not run `docker system prune` or `docker container prune` inside `dind` while conversations are active. These commands delete stopped sandboxes that conversations would resume. Do not run `docker image prune -a`. It deletes Workspace bases. The next new conversation of each Workspace then clones the repository again while a new base builds. ## Recovery runbooks ### Workspace chat fails to start on Docker Compose * A 503 that says the Docker daemon is not reachable means that `dind` is down. Confirm that it is healthy (`docker compose --profile deploy ps -a`). Check `docker compose logs dind`. * A 503 that says the chat sandbox image is missing means that `chat-sandbox-image` has not finished or has failed. For example, a package mirror was unreachable, or someone removed the `sandbox_docker` volume. Check its logs. Then run it again: `docker compose --profile deploy up -d chat-sandbox-image`. * Check backend logs for Docker TLS errors. After the `sandbox_ca` volume is removed, restart `backend` and `worker` so they read the new certificates. * `dind` needs a privileged container. On a host that refuses one, chat is not available. Use the AWS CDK path, or run the sandboxes on another Docker host (see [Chat sandboxes](/docs/self-hosting/configuration#chat-sandboxes)). ### Workspace chat fails to start on AWS * Confirm the sandbox host instance is `InService` in its Auto Scaling group. * Check backend logs for Docker connection or TLS errors. Backend and worker reach the host at `sandbox-host.ctxpipe.local:2376`. The host registers that name when it finishes booting. * If the backend reports that the chat sandbox image is missing, the host could not pull `ghcr.io/ctxpipe-ai/chat-sandbox`. Check the host's outbound access to `ghcr.io` through NAT. * Terminate the instance to get a fresh host. The Auto Scaling group replaces it in a few minutes and reuses the stored TLS certificates. ### App loads but sign-in fails * Confirm `AUTH_BASE_URL` matches the browser origin. * Confirm `AUTH_SECRET` is present and stable. * Confirm provider callback URLs use the public backend origin. * Check backend logs for Better Auth or OAuth provider errors. ### Repository never becomes indexed * Check `codesearch` logs for clone/index errors. * Confirm GitHub App permissions and installation scope. * Confirm `CODESEARCH_URL` points at the codesearch service from the backend. * Reindex after fixing credentials or network access. ### MCP discovery or OAuth fails * Confirm `GET /.status` returns `{ "status": "ok" }`. * Run `npx ctxpipe doctor mcp --url "$AUTH_BASE_URL/mcp"`. * A `ready-for-oauth` result means routing, the Bearer challenge, and OAuth discovery are consistent. It does not prove authenticated `tools/list` or a tool call. * Confirm `AUTH_BASE_URL` matches the public origin clients use, including TLS. * Confirm the MCP URL path is `/mcp`. * For authenticated OAuth, `tools/list`, and a `list_workspaces` call, use a version-pinned MCPJam client against a test organization. See [MCP](/docs/self-hosting/mcp). ### Chat or MCP returns weak context * Confirm repositories are indexed. * Confirm graph extraction has completed for the repository. * Check model provider credentials and rate limits. * Check backend and worker logs around code ingestion and graph projection. ### Knowledge graph fails to load * Confirm `GRAPH_DB_URI` and provider credentials. * Confirm the graph database is reachable from backend and worker. * Check backend logs around graph snapshot/read operations. * Re-run repository ingestion if graph data needs to be rebuilt. ### An index run of the knowledge store takes longer than usual For an organization on the new knowledge store, some events start a full rebuild of a Workspace. An example is a force-push that removed the last indexed commit. A full rebuild reads and embeds each file again, so it takes longer. Chat and MCP continue to answer from the index. See [Changes and full rebuilds](/docs/workspaces/indexing#changes-and-full-rebuilds). The worker log of each run has `knowledgeProjection.mode` (`delta` or `full`) and `knowledgeProjection.fullReason`. The span of the run has the same reason in `ctxpipe.knowledge.full_reason`: `first_run`, `missing_commit`, `projection_version`, `sources_config`, or `attempts`. The linked section gives the cause of each. ### Confluence setup stalls * Confirm Atlassian OAuth callback URL. * Confirm Forge install URL or provisioning token. * Confirm the worker can run Forge provisioning and Confluence sync jobs. * See [Confluence & Atlassian](/docs/self-hosting/confluence-atlassian). ### Linear setup stalls or content becomes stale * Confirm that you registered a Linear OAuth application in the wizard, or that `LINEAR_*` is set on the backend and the worker. * Confirm the Linear webhook secret is saved on the connection or `LINEAR_WEBHOOK_SECRET` is present on backend, and signed requests reach `/api/v1/webhook/linear`. * Confirm the selected GitHub App installation still has write access and that merge pushes reach `/api/v1/webhook/github/`. * Check worker logs for `linear-sync-config`, `linear-sync-content`, or `linear-sync-incremental` failures. * Choose **Sync now** in the menu of the connector card after you fix the underlying credential, network, or permission failure. * See [Linear](/docs/self-hosting/linear). ### PagerDuty setup stalls or incidents become stale * Confirm PagerDuty OAuth credentials are present on backend and worker. * Confirm the connection is live and the incident's service is listed in the scope of a PagerDuty source with the status **Connected**. There is no shared webhook secret. ctxpipe stores the secret of each subscription on the connection. * Confirm signed requests reach `/api/v1/webhook/pagerduty` and merge pushes reach `/api/v1/webhook/github/`. * Check worker logs for `pagerduty-sync-config`, `pagerduty-sync-content`, or `pagerduty-sync-entity` failures. * Choose **Sync now** in the menu of the connector card after you fix the underlying credential, network, or permission failure. * See [PagerDuty](/docs/self-hosting/pagerduty). ## Related pages # Production readiness (https://docs.ctxpipe.ai/docs/self-hosting/production-readiness) Use this checklist before you make a self-hosted deployment available to a team. The reference Compose stack is a first draft. A production deployment must define the network boundary, persistence, secrets, and rollback process. ## Public origin ctxpipe expects a single public backend origin for browser, REST, auth, and MCP traffic. | Check | Why it matters | | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | `AUTH_BASE_URL` is the public HTTPS origin | Better Auth callbacks, generated links, OAuth issuer values, MCP auth, and connector callbacks depend on this value. | | `CTXPIPE_PUBLIC_APP_URL` matches the public backend origin | The UI uses this value at build time. | | Browser users enter through the backend | The backend proxies the UI, keeping auth and API requests on the same origin. | | `AUTH_ALLOWED_ORIGINS` includes only required extra origins | Avoid broad CORS allowances. | ## TLS and proxying Put a TLS-terminating proxy or platform load balancer in front of the backend. Expose only the backend origin publicly unless you have an operational reason to expose other services. | Service | Public exposure | | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `backend` | Public HTTPS | | `ui` | Private; backend proxy should serve it | | `worker` | Private | | `codesearch` | Private | | `postgres` | Private | | `falkordb` or graph database | Private | | `clickhouse` | Private | | `dind` (Docker Compose) or the AWS sandbox host | Private. Do not publish the Docker API or other unauthenticated services on a host that runs chat sandboxes. | The Compose deploy profile binds `ui` (`3002`) and `codesearch` (`3001`) to `127.0.0.1` only. Do not change them to all interfaces. The backend refuses a webhook body above its limit with `413`. The limits are 25 MiB for GitHub, 5 MiB for Linear, Buildkite, and Plain, and 1 MiB for Slack, Notion, PagerDuty, and Atlassian. If your proxy has a lower body limit, set it to at least 25 MiB for `/api/v1/webhook/github`. Set it to 5 MiB for `/api/v1/webhook/linear`, `/api/v1/webhook/buildkite/*`, and `/api/v1/webhook/plain/*`. Telemetry export is outbound only. There is no collector to expose. Set `OTEL_EXPORTER_OTLP_*` to your own endpoint when you want traces, logs, or metrics to leave the app containers. ## Callback URLs Register callback and webhook URLs against your public origin. | Integration | URL shape | | ------------------------------------------ | ------------------------------------------------------------- | | Atlassian OAuth | `{AUTH_BASE_URL}/api/v1/integrations/atlassian/callback` | | Linear OAuth | `{AUTH_BASE_URL}/api/v1/integrations/linear/callback` | | Linear webhook | `{AUTH_BASE_URL}/api/v1/webhook/linear` | | GitHub App webhook | `{AUTH_BASE_URL}/api/v1/webhook/github/` | | Slack OAuth | `{AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback` | | Slack Events API | `{AUTH_BASE_URL}/api/v1/webhook/slack` | | Notion OAuth | `{AUTH_BASE_URL}/api/v1/connectors/notion/oauth/callback` | | Notion webhook | `{AUTH_BASE_URL}/api/v1/webhook/notion?provisioningToken=...` | | PagerDuty OAuth | `{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback` | | PagerDuty webhook | `{AUTH_BASE_URL}/api/v1/webhook/pagerduty` | | Buildkite webhook | `{AUTH_BASE_URL}/api/v1/webhook/buildkite/` | | Plain webhook (ctxpipe creates the target) | `{AUTH_BASE_URL}/api/v1/webhook/plain/` | | MCP | `{AUTH_BASE_URL}/mcp` | | App auth | `/.auth/*` routes under `AUTH_BASE_URL` | ## Persistence Do not treat the reference Compose volumes as disposable in production. On Compose, mount named volumes for Postgres and FalkorDB before the first start. See [Docker: Persistent state](/docs/self-hosting/deployment/docker#persistent-state). | State | Production expectation | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Postgres | Managed backups, restore test, monitoring, and storage alerts | | Graph database | Backup or snapshot according to provider guidance | | ClickHouse (`clickhouse_data` on Compose) | Backup for fast recovery, or a full rebuild from the Workspace repositories (see [Recover the knowledge store](/docs/self-hosting/configuration#recover-the-knowledge-store)). Disk alerts. | | `repo_cache` | Persistent volume if recloning large repositories is expensive | | `zoekt_index_deploy` | Persistent volume if rebuilding indexes is expensive | | Secrets | Stored in a secret manager or platform secret store, not committed files | ## Codesearch capacity Keep `codesearch` at **one replica**. Zoekt shards, git clone cache, and SCIP artifacts are process-local. Extra worker replicas do not raise ingest throughput unless codesearch memory and the injected `OPENWORKFLOW_CONCURRENCY` / `CODESEARCH_INDEXER_CONCURRENCY` / `CODESEARCH_INDEX_PIPELINE_CONCURRENCY` values also grow. Compose `deploy` uses the small pair (indexer 1, pipelines 1, worker concurrency 6). AWS CDK injects the matching pair for `small` / `medium` / `large`. ## Worker memory The worker starts with a Node heap limit of about 55% of the memory its container may use, up to 12 GiB. Native memory, Git, and the sandbox SDKs use the rest. The AWS CDK worker task (1, 2, or 4 GiB for `small`, `medium`, `large`) and a Compose host therefore get a limit that fits their memory. No setting is needed. To use another limit, set `--max-old-space-size=` in `NODE_OPTIONS` on the worker. The worker keeps your value. Keep it below the container memory. The worker also starts with `--heapsnapshot-near-heap-limit=1` and `--diagnostic-dir=/tmp/heap`. If the heap nears its limit, Node writes one `.heapsnapshot` file to `/tmp/heap` before the process exits. The file is lost when the container restarts, unless `/tmp/heap` is on a volume. Writing the snapshot pauses the worker. It needs extra memory and disk space, and the file can be larger than the heap. On a small task the write can fail, and the platform can then stop the process. The result is the same crash as without the snapshot. To turn the snapshot off, set `--heapsnapshot-near-heap-limit=0` in `NODE_OPTIONS` on the worker. The worker logs its heap limit at start (`heapLimitMib` in the `openworkflow.config-loaded` line). ## Secret handling Protect and rotate: * `AUTH_SECRET` * `CONNECTION_SECRETS_ENCRYPTION_KEY`, if set * GitHub App private keys and webhook secrets * Atlassian OAuth client secrets and Forge tokens * Slack client secret, signing secret, and stored bot-token encryption material * Linear OAuth client and webhook secrets, or the shared encryption key used to decrypt connection-stored Linear app secrets * Notion OAuth client and webhook secrets * Model provider API keys * SMTP credentials * OTLP exporter credentials `AUTH_SECRET` and `CONNECTION_SECRETS_ENCRYPTION_KEY` can affect sessions and encrypted connector credentials. Plan each rotation carefully. Verify the connectors that you have after the change. ## Connector readiness Before inviting users: * Create or configure at least one GitHub App connection. * Confirm GitHub webhook delivery reaches the public backend origin. * If you use Confluence, set up Atlassian OAuth and Forge install or provisioning. * If you use Slack, set all three Slack app secrets. Verify the OAuth callback and the Events Request URL. When you need multi-workspace distribution, test the installation in a non-development workspace. * If you use Linear, register a Linear OAuth application. Use the product UI, or `LINEAR_*` on the backend and the worker. * Register the exact OAuth callback and webhook URLs. * Confirm that the backend and the worker can reach `api.linear.app`. * If you use Notion, set `NOTION_CLIENT_ID` and `NOTION_CLIENT_SECRET`. Register the exact OAuth redirect URL. Set `NOTION_WEBHOOK_SECRET` after the first webhook delivery. * If you use PagerDuty, register the exact OAuth callback. * Save the client ID and secret in the connector wizard. Or set `PAGERDUTY_CLIENT_ID` and `PAGERDUTY_CLIENT_SECRET` on the backend and the worker. ctxpipe creates the webhook subscription after authorization. * If you use Buildkite, confirm that the backend and worker can reach `api.buildkite.com`. Confirm that the backend can reach `graphql.buildkite.com`, which the connect check uses. * Confirm that the backend and the worker can reach `identity.pagerduty.com` and `api.pagerduty.com` or `api.eu.pagerduty.com`. * If you use Plain, confirm that the backend and the worker can reach `core-api.uk.plain.com`. Plain posts webhooks only to a public HTTPS URL. Without one, Plain sources use the daily sync and **Sync now**. * Confirm the worker can run connector jobs. * Confirm `codesearch` can clone the expected repositories. * Confirm model provider credentials work for both chat and embedding calls. ## Preflight checklist * `curl https:///.status` returns JSON with `status: "ok"`. * Sign-in works using the enabled auth method. * A user can create or select an organization. * A repository can be added and reaches indexed state. * Chat or MCP can answer a question grounded in the repository. * A Workspace chat answers in a sandbox, and `SANDBOX_PROVIDER` is not `unsandboxed`. See [Operations](/docs/self-hosting/operations) for the sandbox checks. * Backend, worker, and codesearch logs are collected. * Postgres and graph database backups are configured. * Upgrade and rollback steps are written down for the current deployment shape. ## Related pages # Upgrades (https://docs.ctxpipe.ai/docs/self-hosting/upgrades) An upgrade is an operational change. Back up state. Run migrations once. Deploy compatible services together. Validate product workflows before you invite users back in. ## Upgrade checklist 1. Read release notes or the relevant pull request. 2. Back up Postgres. 3. Back up or snapshot the graph database. 4. Keep the current deployment secrets and image tags or commit SHA. 5. Confirm that no connector or ingestion job that runs for a long time must finish first. 6. Run migrations against the target database (AWS CDK: this is part of `cdk deploy`). 7. Deploy backend, worker, UI, and codesearch from the same commit or release. 8. Validate health, sign-in, repository indexing, Chat/MCP, and connectors. ## Upgrade from main to Workspaces This release is a minor upgrade. You change the version number and deploy. The SQL only adds to your database, with one exception. The migration deletes orphan rows that have no organization (see [What the SQL changes](#what-the-sql-changes)). Nothing runs on its own after the deploy. Each change to a Workspace repository and each paid run waits for a click in the UI. Hosted and self-hosted deployments use the same UI workflow. ### Before you start 1. **Take a database snapshot.** The upgrade deletes orphan rows. A schema rollback is possible only from a snapshot. Take the snapshot of Postgres (and of the graph database) before you deploy. 2. **Check the GitHub App permissions.** Workspace writes need **Contents: Read and write** and **Pull requests: Read and write**. Without them, a Workspace shows `blocked: write_access`. The upgrade pull request then cannot open. Update the permissions in the GitHub App settings. Then accept the change for each installation. 3. **Note your current image tag or release.** You need it for a rollback. ### Deploy Use the flow for your platform below. Every platform runs the same single migrate command, `bun run apps/backend/src/db/migrate.ts`. * **Docker Compose:** pull the new release, then run `docker compose --profile deploy build` and `docker compose --profile deploy up -d`. The `migrate` service runs before the app services. `chat-sandbox-image` builds the new chat image in `dind`. Set `POSTGRES_PASSWORD` in your `.env` (Compose sends it to migrate as the `ctxpipe_app` password). Check `docker compose logs migrate`. Before the Workspaces release, the deploy profile published Postgres on host port 5433 (`CTXPIPE_POSTGRES_HOST_PORT`). From the Workspaces release, it publishes no Postgres port. If a backup job or a tool on the host uses port 5433, change it to `docker compose exec postgres pg_dump -U ctxpipe ctxpipe > backup.sql`. * **AWS CDK:** run `pnpm update @ctxpipe/aws-cdk`, then `cdk deploy`. See [the sandbox host](#aws-cdk-sandbox-host) for the cost of the new host. * **Image-based:** run migrate once with the owner `DATABASE_URL`. Pass the app password as `DATABASE_APP_PASSWORD`, or make sure the `ctxpipe_app` role exists. Then start the new images with the `ctxpipe_app` URL. ### What the SQL changes The upgrade follows the expand-only rule. A minor release adds tables, columns, policies, and rows. It does not remove anything that the previous release reads. * The migration deletes orphan rows (rows with no `org_id`) in `claim_evidence`, `confluence_spaces`, and `repository_checkouts`. The delete runs in the same transaction as the migration. If the migration fails, the rows stay. ctxpipe keeps no copy of these rows. * `org_id` stays nullable on these tables. Row-level security hides rows with no `org_id` from the app role. * A `BEFORE INSERT` trigger on each of the three tables copies `org_id` from the parent row when it is null. This covers writes from the previous release after a rollback. * Every migrate run then calls the backfill, which repairs the other rows that have no `org_id`. The backfill is safe to run again. You do not run it by hand. * The only new column is the nullable `workspaces.format_version`. It caches the format of the Workspace repository. Git stays the source of truth. ### Removed routes The old repository list, chat, and knowledge graph pages are gone. A bookmark to `//repositories`, `//chat`, or `//knowledge-graph` opens the organization home page. The same data is in the Workspace. ### Connector handover After the upgrade, each connector binding that a Workspace owns stops writing to the old repository. The Linear, Notion, Slack, and Confluence connectors write only to the Workspace. No connector writes the same content to both places. You do not need to change a connector setting. ### Workspace format file Each Workspace repository holds `.ctxpipe/workspace.yaml`. It has one key, `format`, with an integer. ctxpipe reads it at every hydrate and caches it in `workspaces.format_version`. | Format | Layout | | ------ | ------------------------------------------------------------------------------------------- | | 0 | No file: a repository from before Workspaces, or a foreign repository | | 1 | Knowledge that ingestion cannot rebuild is exported; sources are in `.ctxpipe/sources.yaml` | | 2 | One Markdown file for each source under `sources/` | | 3 | Ingestion v2 ownership front matter (current) | * Only an upgrade pull request or the bootstrap commit of a new Workspace writes the file. Bootstrap writes it on an empty repository, or on a repository with no `.ctxpipe/`, `sources/` or `repositories/` folder below the Workspace root folder. No agent or chat write can change it. * A file that does not parse makes the Workspace read-only (`format_invalid`). Reads still work. * A format above the one your release knows (for example after a rollback) makes the Workspace read-only (`format_ahead`). Reads still work. ### Upgrade a Workspace An upgrade of a Workspace has two parts. These are a git upgrade and a re-ingestion. Each part needs one click. Nothing starts by itself, unless an owner or admin turns on [automatic upgrades](/docs/workspaces/settings#automatic-upgrades). A merge always needs a person. This is true on hosted and on self-hosted deployments. #### Where to find the upgrades * **Workspace settings:** open the Workspace, then **Settings**. The **Status** group lists each open item. It also shows the format of the repository, the format that ctxpipe reads, and the automatic-upgrades setting. * **Home alert:** the organization home page shows one alert for each Workspace that has an item to finish. Click **Review upgrade** to open the Workspace settings. The alert shows nothing when no item is pending. #### Who can start an item An organization owner or admin can click **Start**. A member sees the upgrades but gets **Ask an admin** in place of the button. #### Item status Each item shows one status. For the statuses, the next action for each status, and the reasons for a blocked item, see [Upgrades](/docs/workspaces/settings#upgrades) in Workspace settings. If someone closes the pull request without a merge, the item goes back to **Ready to start** with a note. Click **Start** to open a new pull request. #### Step 1 to 3: the git upgrade 1. Click **Start** on the git upgrade. ctxpipe opens one pull request for the Workspace on the branch `ctxpipe/upgrade/format-`. ctxpipe never pushes to the default branch. 2. The pull request holds one commit for each step that changes a file. The last commit writes `.ctxpipe/workspace.yaml`, the format file. This is the format commit. 3. Review the pull request. Then merge it. The merge lands the files and the new format together. The steps run in this order: | Step | Result | | ---- | -------------------------------------------------------------------------------------------------------------------------------- | | 1 | Moves the sources into `.ctxpipe/sources.yaml`. Exports the knowledge that fresh ingestion cannot make again. | | 2 | Writes one Markdown file for each source under `sources/`. | | 3 | Gives the files that the old pipeline extracted to ingestion v2. Writes the changes that hydrate makes on a format 3 repository. | **What step 1 keeps.** Step 1 writes this knowledge to the `knowledge/` folder of the Workspace repository: * Knowledge from a repository that is not on GitHub. * Knowledge from a GitHub repository that the sources file does not link. * Knowledge of the Workspace repository that no Markdown file of the Workspace holds. * The teams from CODEOWNERS files. **What the report lists.** Step 1 does not export other knowledge. The re-ingestion or hydrate makes it again, or no source reads it. The pull request description is the report. It lists each item that a step kept, added, or could not move, for example a connection that maps to no Workspace. Read the report before you merge. **A merge is a second approval.** The click on **Start** approves the pull request. The merge approves the change to your repository. Without a merge, the format stays the same. #### Re-ingestion Fresh ingestion from your source repositories is a standard upgrade step. It runs ingestion v2 again for each GitHub source repository of the Workspace. Until you merge the pull request and run it, MCP serves only what the Workspace already holds. * **Needs format 3.** The item is blocked until the Workspace is at format 3. Merge the git upgrade first. * **Throttle.** One re-ingestion of an organization runs at a time. Its repositories run one after the other. An instance runs at most four re-ingestions at once. A run that waits for its turn stays queued and starts when a slot is free. See [sandbox limits](/docs/workspaces/chat#sandbox-limits). * **Spend.** ctxpipe records the model spend of each repository run. It does not cap the spend. A run does all of its steps. Use the limits of your model provider key to control spend. When a runtime reports no usage for a model step, the report shows "Spend unknown". The spend then counts only the known part. * **Restart and resume.** A run that a restart of the worker interrupts continues. If a run stops with no result, a new click on **Start again** makes a new run. A repository that an earlier run finished at the same commit is not paid for again. * **Cancel.** An owner or admin can stop a queued or running re-ingestion with **Cancel** on its row, or with `POST /{orgSlug}/api/v1/workspaces/{workspaceSlug}/upgrades/{key}/cancel`. ctxpipe stops the upgrade run, its repository runs and their ingestion runs. They start no next step, so they make no new model call and publish nothing. A step that runs at the cancel can end, but it records nothing. A publish step pushes only while its run is not canceled, and a cancel waits until a push that started ends. The item becomes **Canceled** and gets no automatic retry. **Start again** makes a new run. An organization with GitHub data but no Workspace is a new user. It creates its Workspace in normal onboarding, and its repositories get a fresh ingestion there. ### Repositories from before Workspaces Before Workspaces, an organization linked code repositories to the organization. When the organization moves to Workspaces, ctxpipe links these repositories to its **first Workspace**. The upgrade pull request of that Workspace does the work: * A GitHub repository becomes a GitHub source. After step 2 it is a source file at `sources/github//.md`. * A `repositories/*.md` file of a GitHub repository becomes a source file too. Its `branch` becomes the `ref`. ctxpipe deletes the `repositories/*.md` file in the same pull request. * A repository that the Workspace repository already is, or that another source already links, is not added again. * A repository that has no GitHub connection becomes a plain git source file at `sources/git//.md`, when ctxpipe can read its URL as a plain git URL under the deployment's URL rule. See [Plain git repositories](/docs/workspaces/linked-repositories#plain-git-repositories) for the URL rules. * ctxpipe does not link any other repository. The pull request description names each one. To add a plain git repository as a source, open a pull request in Git that adds its source file. Workspace **Settings** lists only GitHub App repositories. Or leave the repository out. Below format 3, the Workspace serves the knowledge from before Workspaces as read-only units from the database (source `legacy`). After the re-ingestion of a repository, the fresh knowledge replaces the old knowledge of that repository. #### Index a repository with the REST API The repository API is from before Workspaces. It indexes a repository for the organization: * `POST /{orgSlug}/api/v1/repositories` with `name` and `gitUrl` adds a repository and queues its indexing. A plain git repository works the same way. * `POST /{orgSlug}/api/v1/repositories/{id}/reindex` starts its indexing again. Each index of a repository also updates the knowledge of each Workspace whose source file names it. A repository is part of a Workspace only when a source file of that Workspace names it. ### AWS CDK sandbox host The chat sandbox host is **on by default**. A deploy of this release adds one EC2 host. The default instance for `size: "small"` is `t4g.medium` (2 vCPU, 4 GiB, 30 GB Docker volume). It costs about **$28 a month** in us-east-1. That is the bottom of the $28 to $107 range of the three `size` profiles. It runs about 6 active sandboxes at once, which is the smallest instance that meets the sandbox needs. See [the sandbox host](/docs/self-hosting/deployment/aws#chat-sandbox-host) for the larger sizes. To turn the host off, set `sandboxHost: { enabled: false }` on `CtxPipe`. The deploy then adds no EC2 host and gives the services no sandbox host settings. Without the host, chat is off. It answers "sandbox unavailable" (503). ## AWS CDK flow To upgrade a self-hosted AWS stack, update `@ctxpipe/aws-cdk` and deploy again. The published package pins the backend, worker, UI, codesearch, and migrate images to the same git SHA. It also applies the Fargate size profiles of that release. Before the first deploy of a new version, read the Aurora PostgreSQL minor version of your cluster. CloudFormation cannot move a cluster to a lower minor version. A cluster above the pinned version therefore needs that version set in your CDK app. See [Aurora PostgreSQL minor version](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#aurora-postgresql-minor-version). ```bash pnpm update @ctxpipe/aws-cdk cdk synth cdk deploy ``` `cdk deploy` runs the migrate task. Then it updates the ECS services. Do not retag services to `:latest` or mix image SHAs from different package versions. See [AWS](/docs/self-hosting/deployment/aws). The migrate task of this release also creates the `ctxpipe_app` runtime role. The services then read a new runtime secret with the `ctxpipe_app` URL. The earlier `DATABASE_URL` secret keeps the owner URL, so a rollback to the earlier task definitions still works. You do not run `psql` or add `CtxPipe` props. Image-only retags without a package bump skip that role split. ### AWS CDK knowledge store host The first upgrade to a version with the knowledge store adds one EC2 host for ClickHouse, an EBS data volume, and an S3 bucket. The host is always on: no prop turns it off. At `size: "small"` it is a `t4g.large` with a 50 GB volume, for about **$55 a month** in us-east-1. The deploy waits until ClickHouse answers. Then the migrate task makes its tables and its two users, before the services roll. The new knowledge store stays off for each organization. See [the knowledge store](/docs/self-hosting/deployment/aws#knowledge-store) for the larger sizes and for recovery. ## Docker Compose flow For source-based Compose deployments, keep your `docker-compose.override.yml` (named volumes, extra environment variables, port bindings) in place, then run: ```bash git fetch origin git checkout docker compose --profile deploy build docker compose --profile deploy up -d ``` `docker compose --profile deploy up -d` also runs `chat-sandbox-image` again. If the chat image changed, it builds the new image inside `dind`. Then each conversation gets a new sandbox at its next turn, and continues from its git branch. The `deploy` profile runs the `migrate` service before app services. Migrate connects as the Postgres owner (`ctxpipe` by default) and provisions `ctxpipe_app`. Backend, worker, and codesearch use the app-role `DATABASE_URL`. Check the migrator logs before you validate the app. ```bash docker compose logs migrate docker compose logs backend worker codesearch ``` ## Image-based flow For platforms that run prebuilt images: 1. Build images for backend, worker, UI, and codesearch from the same commit. 2. Push images with an immutable tag, such as a commit SHA. 3. Run migrations using the backend image for that same commit, with the **owner** Postgres URL. Then point backend, worker, and codesearch at the `ctxpipe_app` URL (AWS CDK does this inside `cdk deploy`). 4. Update all app services to the new image tags. 5. Keep the previous image tags available until validation completes. ## Validation After deployment: | Area | Validation | | -------------- | -------------------------------------------------------------------------------------------------------------------- | | Backend | `GET /.status` returns `status: "ok"` | | Auth | Sign in with the configured provider | | UI | Open app through `AUTH_BASE_URL`; do not bypass backend proxy | | Repositories | Existing repository list loads; new or reindexed repository succeeds | | Codesearch | Search and file retrieval work for an indexed repository | | Graph | Knowledge graph page loads for an organization with indexed data | | MCP | MCP client connects using OAuth | | Workspace chat | A conversation answers, and its sandbox shows on the Docker daemon (see [Operations](/docs/self-hosting/operations)) | | Connectors | GitHub webhooks, scoped mirrors, and Slack thread capture still work | | Observability | Logs and traces continue to reach your configured sinks | ## Rollback Rollback depends on whether the upgrade ran database migrations. | Situation | Rollback approach | | --------------------------------- | ----------------------------------------------------------------------------------------------- | | No migrations ran | Revert service images or source ref and restart services. | | Forward-compatible migrations ran | Revert services only if the previous version can tolerate the new schema. | | Breaking migrations ran | Restore Postgres from backup, restore graph data if needed, then redeploy the previous version. | ### Roll back from Workspaces to main The SQL of this release is expand-only, so the previous (`main`) image can run on the upgraded database. Do all of these steps: 1. Put back the previous image tag on every runtime service. 2. **Put back the owner `DATABASE_URL`** on every runtime service. The `main` image does not set `app.organization_id`. On the `ctxpipe_app` role, row-level security hides all organization rows, so the app sees no data. On AWS CDK, deploy the previous `@ctxpipe/aws-cdk` version and restore the previous runtime secret. 3. Start the services. After a rollback, the `main` image writes `repository_checkouts`, `confluence_spaces`, and `claim_evidence` rows with no `org_id`. A trigger fills `org_id` from the parent row, and the next migrate run repairs the rest. You do not need to do anything for these rows. The deleted orphan rows do not come back. Only the snapshot you took before the upgrade holds them. It is also the only way to roll back the schema. A Workspace that you upgraded keeps its new format. A release that knows only older formats shows that Workspace as read-only (`format_ahead`). A code rollback without a database restore is not always safe. Take a backup before you run migrations. Test the restore in an environment like your production environment. ## Upgrade notes by component | Component | Watch for | | -------------- | -------------------------------------------------------------------------------- | | Backend | Schema migrations, auth behavior, MCP route compatibility, connector API changes | | Worker | Workflow compatibility, connector job changes, Forge provisioning changes | | UI | Build-time public URL values, connector wizard behavior | | Codesearch | Index format or clone/cache behavior | | Chat sandboxes | Chat image changes, Docker daemon and TLS settings, sandbox host size (AWS) | | Graph database | Provider configuration and graph projection compatibility | | Model provider | Default model names, embedding dimensions, provider-specific auth | ## Related pages # Buildkite (https://docs.ctxpipe.ai/docs/self-hosting/buildkite) ## Overview A self-hosted ctxpipe uses the same Buildkite connector flow as hosted ctxpipe. Buildkite has no OAuth application for other products. An organization owner or admin pastes a Buildkite API access token in the setup. The webhook token is also connection-specific. ctxpipe encrypts both tokens on the connection with the connection secret key of the deployment. There is no `BUILDKITE_*` environment variable. You do not need to change Compose or CDK settings. ## What you need * A public HTTPS `AUTH_BASE_URL` for the webhook. Without the webhook, the connector syncs once a day and on **Sync now**. * A ctxpipe GitHub App connection with write access to each Workspace repository that syncs Buildkite builds. * Network access from the backend and the worker to `api.buildkite.com` over HTTPS. The backend also reaches `graphql.buildkite.com`, only when someone connects. ## Webhook URL Each connection has its own webhook URL, because each webhook has its own token: ```text {AUTH_BASE_URL}/api/v1/webhook/buildkite/{connectionId} ``` The webhook step of the setup shows the full URL. ctxpipe verifies the `X-Buildkite-Signature` HMAC-SHA256 of the timestamp and the raw request body, and refuses a signature older than five minutes. It also refuses a webhook that sends the plain `X-Buildkite-Token`. A failed enqueue returns `503`. The next event in the window or the daily sync then gets the build. All traffic stays on your deployment. Buildkite sends the webhook to your `AUTH_BASE_URL`. Your backend and worker call `api.buildkite.com`. When someone connects Buildkite, your backend also calls `graphql.buildkite.com` once. ## Verify the connector 1. Open **Connectors**, select **Add connection**, then select **Connect** on the **Buildkite** row. 2. Paste an API access token with only `read_pipelines` and `read_builds`. Paste the organization slug. ctxpipe refuses a token with any other scope or with GraphQL API access. 3. Add the webhook in Buildkite with the signature option and the `build.finished` event, then save its token in the setup. 4. In **Add Buildkite to Workspaces**, select a Workspace and at least one pipeline. Merge the pull request that adds the source file under `sources/buildkite/`. 5. Confirm that the source shows **Connected** in the Workspace settings and files appear under `buildkite/pipelines//`. 6. Finish a build in a selected pipeline. Confirm that `POST /api/v1/webhook/buildkite/{connectionId}` returns `200`. Confirm that the build file appears after the 10-minute window. ## Troubleshooting ### Builds do not arrive * Confirm that Buildkite can reach `AUTH_BASE_URL` over public HTTPS. With a private URL, no webhook arrives, and the connector syncs once a day and on **Sync now**. * A `401` answer has three causes: * The webhook sends the plain `X-Buildkite-Token`. * The token in the setup is not the token of this webhook. * The clock of the backend is more than five minutes off. Select the signature option in Buildkite. Paste the webhook token again in the setup. Check the time of the backend host. * A `503` answer means that ctxpipe could not queue the sync. Read the backend logs. The next event in the window or the daily sync gets the build. ### The setup cannot check GraphQL access The setup shows "ctxpipe cannot make sure that the token has no GraphQL API access. Try again later." when the backend cannot reach `graphql.buildkite.com`. Allow outbound HTTPS from the backend to that host, then connect again. See [Buildkite connector](/docs/connections/source-connectors/buildkite) for the files, the sync, and product troubleshooting. # Confluence & Atlassian (https://docs.ctxpipe.ai/docs/self-hosting/confluence-atlassian) ## Overview Self-hosted Confluence uses the same connector surface as hosted ctxpipe. But your deployment owns the Atlassian and Forge configuration. There are two modes: * **Managed-style install URL**: give a Forge install URL for the connection or deployment. The wizard then opens Atlassian's install flow. * **Self-hosted provisioning**: the worker creates, deploys, and installs a Forge app. It uses a scoped Forge API token and the Confluence site hostname. Both modes keep connector settings and encrypted credentials on your deployment. ## What you need * A public `AUTH_BASE_URL` or `CTXPIPE_PUBLIC_APP_URL` that Atlassian can reach. * A Confluence Cloud site, for example `your-site.atlassian.net`. * A GitHub connection, and a Workspace that the GitHub App can write to. * For per-connection OAuth: an Atlassian 3LO app. * For provisioning: an Atlassian account email and Forge scoped API token. ## 1. Register Atlassian OAuth If your deployment sets global **`ATLASSIAN_CLIENT_ID`** and **`ATLASSIAN_CLIENT_SECRET`**, ctxpipe uses that Atlassian app for account linking. Otherwise, the wizard shows **Register Atlassian OAuth app**. Register a 3LO app for the Forge connection in the [Atlassian developer console](https://developer.atlassian.com/cloud/oauth-2-3lo-apps). Save its client ID and secret in that step. On Docker Compose, the `ATLASSIAN_*` values need an override file (see [Docker](/docs/self-hosting/deployment/docker#settings-that-need-an-override-file)). On AWS CDK, use `connectorSecrets.atlassianClientId` and `connectorSecrets.atlassianClientSecret`. Use this callback URL: `{AUTH_BASE_URL}/api/v1/integrations/atlassian/callback` ctxpipe encrypts per-connection credentials and does not return the client secret from connector metadata. ## 2. Install or provision Forge The wizard resolves the Forge install path in this order: 1. An install URL saved for this Confluence connection. 2. Deployment-wide **`CONFLUENCE_FORGE_INSTALL_URL`**. 3. No install URL, which shows the self-hosted provisioning form. Provisioning runs `forge register`, `forge deploy`, and `forge install` through a durable worker job. The generated Forge app uses your public API origin for its remote callback. The Forge app requires `read:attachment:confluence` to clone page attachments. After you upgrade the app manifest that you have, deploy the new Forge version. Complete Atlassian's permission upgrade. A refresh of the app-system token alone does not grant the new scope. ctxpipe stores assets in the Workspace repository. The limits are 25 MiB for each file and 100 MiB for each page. The worker removes credentials from media redirects. It never saves Atlassian download URLs. ### Allowed Forge apps ctxpipe accepts a Forge call only from the Forge app of that Confluence connection. The allowed app is one of these: * The app ID that the connection saved. Provisioning saves this ID before `forge install`. * When the connection has no saved app ID: the app ID in the install URL of the connection, the app ID in **`CONFLUENCE_FORGE_INSTALL_URL`**, or the hosted ctxpipe app. The first install event saves the ID of the app. ## 3. Finish the connector After Atlassian is linked and Forge is installed: 1. Link or confirm the GitHub connector. 2. In **Add to Workspaces**, select the Workspaces that receive Confluence content. 3. Select the Confluence spaces and pages to ingest. 4. Review and merge the pull request that adds `sources/confluence/.md` in each Workspace repository. ctxpipe then syncs Confluence content into each selected Workspace repository. The content goes through the same indexing and graph ingestion path as other source material. ## Error codes (provision) | Code | What to check | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `forge_missing_operator_email` | Give the email of the Atlassian account that owns the Forge token | | `forge_auth_failed` | Forge scoped API token (App: Forge) at id.atlassian.com | | `forge_developer_space_noninteractive` | The Atlassian account has no Forge developer space, and the job cannot create one. Create it in the Atlassian developer console, then retry. | | `forge_developer_space_ensure_failed` | ctxpipe could not find or create the Forge developer space. Retry; contact support with job id | | `forge_lint_failed` | Bundled app; contact support with job id | | `forge_deploy_forbidden` | Atlassian user cannot deploy this app | | `confluence_install_forbidden` | Install requires Confluence admin | | `confluence_install_site` | Site host / Confluence product | | `network` | Firewall, DNS, outbound HTTPS | | `unknown` | Retry; contact support with job id | ## Environment (operators) * **`AUTH_BASE_URL` / `CTXPIPE_PUBLIC_APP_URL`** - public origin for API and callback URL registration. * **`ATLASSIAN_CLIENT_ID` / `ATLASSIAN_CLIENT_SECRET`** (optional) - global Atlassian OAuth app for account linking. If unset, use per-connection 3LO in the wizard. * **`CONFLUENCE_FORGE_INSTALL_URL`** (optional) - deployment-wide fallback for the **Install** step when a connection does not have its own install URL. ## Related pages # GitHub (https://docs.ctxpipe.ai/docs/self-hosting/github) GitHub is the base of a self-hosted ctxpipe deployment. The GitHub App reads the linked code repositories. It also writes to the Workspace repositories. It writes the knowledge, the source files, and the content of Linear, Notion, Confluence, PagerDuty, and Slack. Your deployment must use a GitHub App that **you create and own**. Self-hosted repository traffic and credentials do not pass through the hosted ctxpipe GitHub App. ## Before you start * Configure a stable public HTTPS `AUTH_BASE_URL`. * Sign in as an owner or admin of the ctxpipe organization. * Decide if a GitHub user account or a GitHub organization owns the App. * Make sure that you can install the App on the GitHub accounts that own the target repositories. A GitHub OAuth App used for product sign-in is not the repository GitHub App. The repository App has its own ID, private key, webhook secret, permissions, and installation. ## 1. Start the in-product setup Open **Connectors**, select **Add connection**, then select **Connect** on the **GitHub** row. The self-hosted wizard reserves a connection and shows the exact callback, setup, and webhook URLs for that connection. Keep the wizard open while you register the App. The webhook URL contains a connection identifier. Copy it from the wizard. Do not guess it. ## 2. Register the GitHub App Create a [new personal GitHub App](https://github.com/settings/apps/new), or open a GitHub organization's **Settings → Developer settings → GitHub Apps → New GitHub App**. Use the values that the ctxpipe wizard shows: | GitHub field | Value | | ---------------------- | --------------------------------------------------- | | **GitHub App name** | A unique name your operators recognize | | **Homepage URL** | Your `AUTH_BASE_URL` | | **Callback URL** | `{AUTH_BASE_URL}/.auth/api/v1/auth/callback/github` | | **Setup URL** | `{AUTH_BASE_URL}/.github/setup` | | **Redirect on update** | Enabled | | **Webhook active** | Enabled | | **Webhook URL** | Copy the connection-specific URL from the wizard | | **Webhook secret** | Copy the generated secret from the wizard | The connection-specific webhook follows this shape: ```text {AUTH_BASE_URL}/api/v1/webhook/github/ ``` ## 3. Grant the required permissions Set repository permissions: * **Contents:** Read and write * **Pull requests:** Read and write * **Issues:** Read-only Set account permissions: * **Email addresses:** Read-only Subscribe to these events: * **Pull request** * **Pull request review** * **Pull request review comment** * **Issues** * **Issue comment** (conversation on issues and pull requests) * **Push** * **Repository** The pull request events let ctxpipe mirror the closed pull requests of every linked repository, merged or not. It writes them into `github/pulls///` of each Workspace that links the repository. The **Issues** and **Issue comment** events let ctxpipe mirror the issues of every linked repository into `github/issues///` in the same way. The App installation must be able to read those linked repositories. ctxpipe does not mirror required-check conclusions. A backfill of the issues of one linked repository reads up to 200 of the most recently updated issues. It sends one GraphQL request for each 50 issues. Each request includes the 100 most recent comments of each issue. It writes all the files in one commit. ### Add the Issues permission to an existing App An App that you created before the issue mirror does not have **Issues: Read-only** or the **Issues** event. Do these steps: 1. In the GitHub App settings, set **Issues** to **Read-only** under repository permissions. 2. Subscribe to the **Issues** event. 3. Save the changes. 4. Tell the owner of each installation of the App to accept the new permission. GitHub shows the request on the installation settings page of that account and sends an email to the owner. Until the owner of an installation accepts the permission, ctxpipe does not mirror the issues of that installation. The pull request mirror and indexing continue to work. When the owner accepts, GitHub sends an `installation` event. ctxpipe then backfills the issues of each linked repository of that account. After that, ctxpipe mirrors each issue change. ctxpipe needs write access because it commits to Workspace repositories. It also opens pull requests for source changes, upgrades, and Workspace chat. Limit the App installation to the repositories that ctxpipe must read or write. ## 4. Generate and save credentials After you create the App: 1. Copy its numeric **App ID**. 2. Copy the App slug from its public URL. 3. Generate and download a private key (`.pem`). 4. Return to the ctxpipe wizard. 5. Paste the App ID and slug, then upload or paste the complete PEM. 6. Save the connection. ctxpipe encrypts the private key and webhook secret for this GitHub connection. Keep `AUTH_SECRET` stable. Use `CONNECTION_SECRETS_ENCRYPTION_KEY` when your secret-management policy needs an explicit connector encryption key. ## 5. Install and verify the App Install the App on the GitHub user or organization that owns the source repositories. Choose **Only select repositories** unless the deployment needs broad access. ctxpipe attaches an installation to an organization only with proof that it belongs there. A connection that you create in the wizard gives this proof with its own App credentials. An installation of an App that you configure with `GITHUB_APP_ID` is different. The person who finishes setup must show that their GitHub account can see the installation. If their linked GitHub account cannot show this, ctxpipe asks them to authorize the App once. ctxpipe uses the short-lived token of that authorization for this check only. It does not keep the token and does not link the GitHub account. The authorization needs the client ID and the client secret of that same App in `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET`. These variables also turn on GitHub sign-in. The App must list the callback URL `{AUTH_BASE_URL}/.auth/api/v1/auth/callback/github`. Without these values, use the wizard. Then verify the full path: 1. The GitHub connector reaches **Connected**. 2. **Create workspace** lists repositories from the expected installation. 3. A linked repository reaches **Indexed** in the Workspace's **Settings**. 4. Changing repository selection returns through the ctxpipe setup URL. 5. A test push reaches the connection-specific webhook. 6. A source change can open a pull request on the Workspace repository. ## Troubleshooting ### Installation returns to the wrong host Check the App's Callback URL and Setup URL against the current `AUTH_BASE_URL`. Keep browser, auth, callbacks, API, and MCP on the same public backend origin. ### Setup asks to connect a GitHub account ctxpipe attaches the installation through the App that `GITHUB_APP_ID` configures. Link a GitHub account that can see the installation, or create the connection with the self-hosted wizard so its own App credentials prove the installation. ### Repositories do not appear Confirm that the App is installed on the correct GitHub account and has access to the repository. A change to the installation in GitHub does not tell ctxpipe to index a repository. After that, link the repository from the Workspace's **Settings**. ### Webhooks fail signature verification The secret in the GitHub App settings must match the secret saved for this ctxpipe connection. Do not reuse a connection-specific URL with credentials from another connection. ### Issues do not appear in the Workspace Confirm that the App has **Issues: Read-only** and that it subscribes to the **Issues** event. Then find the cause in the backend logs: * `github_issue_mirror_no_permission`: the owner of the installation did not accept the Issues permission. Tell the owner to accept it on the installation settings page in GitHub. ctxpipe then backfills the issues. * `github_issue_mirror_repository_not_covered`: the installation does not have access to the repository. Add the repository to the repository access of the installation in GitHub. After that, ctxpipe mirrors each issue change of that repository. ### Connector writes fail Confirm that **Contents** and **Pull requests** are both read and write. Confirm that the installation includes the Workspace repository. # Linear (https://docs.ctxpipe.ai/docs/self-hosting/linear) Self-hosted Linear uses an OAuth application that you create in Linear. The usual path is to paste the client ID, client secret, and webhook signing secret into the connector wizard. That path needs no `LINEAR_*` environment variables and no service restart. The same routes run on hosted ctxpipe and on self-host. Hosted ctxpipe keeps a deployment-wide Linear app in environment variables as a fallback. Self-host operators can use that fallback too. Use it when you want one shared Linear application for the whole deployment. The worker decrypts the saved client secret with the same key the backend uses (`AUTH_SECRET` or `CONNECTION_SECRETS_ENCRYPTION_KEY`). That is not a new Linear environment variable. ## What you need * A public HTTPS `AUTH_BASE_URL`. In production, keep `CTXPIPE_PUBLIC_APP_URL` on the same origin. * Permission to create a Linear OAuth application. * A ctxpipe GitHub App connection with write access to the repository of each Workspace that uses Linear. * Network access from the backend and the worker to `api.linear.app` over HTTPS. ## Connector state and backups You can review the Linear scope in Git: * Each Workspace has a source file at `sources/linear/.md`. Draft scope is the file on the branch of the pull request that adds it. * Live scope is the same file on the default branch after the merge. * The deployment stores encrypted credentials and setup status on the `connections` row. The connection holds authorization only. * Signed Linear events update in-scope generated files through durable worker jobs. Back up PostgreSQL for connection credentials. Back up the Workspace repositories for both scope and mirrored content. PostgreSQL contains only credentials and setup state. ## 1. Create a Linear application Open [Create new application](https://linear.app/settings/api/applications/new) in Linear Settings → API. Fill in **Application name** and **Developer name**. In **Redirect URIs**, paste: ```text https:///api/v1/integrations/linear/callback ``` Linear supports several redirect URIs, one per line. Add local, staging, or preview callbacks individually when those environments must run OAuth. The scheme, host, path, and trailing slash must match the URI that ctxpipe sends. ctxpipe requests the **`read`** scope with a user actor on the authorize URL. You do not set those on this Linear form. Linear cannot deliver webhooks to a private localhost address. To test a connector end to end, use a public HTTPS deployment or tunnel. ## 2. Configure webhooks on the same form Turn **Webhooks** on and set **Webhook URL** to: ```text https:///api/v1/webhook/linear ``` Copy **Webhook signing secret** from that form. Linear fills it in before you select Create. ctxpipe verifies the `Linear-Signature` HMAC against the raw request body. It rejects stale timestamps. For an accepted in-scope event, ctxpipe starts a durable update before the endpoint acknowledges delivery. Under **Data change events**, select Issues, Comments, Documents, Projects, Initiatives, Teams, and the related update and attachment boxes (including cycles, issue labels, users, and customer requests). Under **Authorization events**, select OAuth authorization events. Then select **Create**. Only after that, Linear shows **Client ID** and **Client secret**. Copy them for the next step. A Linear application has one webhook destination. You can register several OAuth redirect URIs, but events from every workspace installation go to that one webhook URL. Use a separate Linear application when a staging or PR environment must receive its own incremental events. Per-organization apps each register the same Webhook URL and store their own signing secret on the connection. ## 3. Register the app in the product UI 1. Open **Connectors**, select **Add connection**, then select **Connect** on the **Linear** row. 2. On **Register Linear OAuth app**, paste the client ID and the client secret that Linear shows after Create. Also paste the webhook signing secret from the create form. The wizard shows the exact **Redirect URIs** and **Webhook URL** values for this deployment. 3. Save the app, then continue to **Connect Linear**. 4. Complete OAuth. In **Add to Workspaces**, choose the Workspaces and the scope, then merge the pull request that adds `sources/linear/.md`. ctxpipe stores secrets encrypted on the Linear `connections` row. The list and status APIs never return them. The backend and worker must share the encryption key (`AUTH_SECRET` / `CONNECTION_SECRETS_ENCRYPTION_KEY`) so token refresh can use the saved client secret. ## 4. Optional deployment-wide environment variables `LINEAR_*` is an optional operator shortcut for a deployment-wide Linear application (the hosted fallback). When you set both `LINEAR_CLIENT_ID` and `LINEAR_CLIENT_SECRET`, the wizard skips Register and starts at Connect. | Variable | Required | Purpose | | ----------------------- | ---------------------------------: | ------------------------------------------------------------------------------------------ | | `LINEAR_CLIENT_ID` | Optional; backend + worker if used | Shared OAuth application client ID | | `LINEAR_CLIENT_SECRET` | Optional; backend + worker if used | Shared OAuth code exchange and token refresh | | `LINEAR_REDIRECT_URI` | Backend; optional | Exact callback override; defaults to `{AUTH_BASE_URL}/api/v1/integrations/linear/callback` | | `LINEAR_WEBHOOK_SECRET` | Optional; backend if used | Shared signing secret for `/api/v1/webhook/linear` | Empty values still parse as unset. Restart or redeploy both services after changing these variables. ### Docker Compose Add the variables to the root `.env` used by the `deploy` profile only if you want a deployment-wide app. Then run: ```bash docker compose --profile deploy up -d --build backend worker ``` ### AWS CDK Pass the four values through `connectorSecrets` when you want the deployment-wide fallback: ```ts new CtxPipe(stack, "CtxPipe", { // other required settings connectorSecrets: { linearClientId: cdk.SecretValue.secretsManager("linear/client-id"), linearClientSecret: cdk.SecretValue.secretsManager("linear/client-secret"), linearWebhookSecret: cdk.SecretValue.secretsManager("linear/webhook-secret"), // Optional when AUTH_BASE_URL already produces the registered callback: linearRedirectUri: cdk.SecretValue.unsafePlainText( "https://app.example.com/api/v1/integrations/linear/callback", ), }, }); ``` The construct stores these values in the connector secret and injects them into backend and worker task definitions. ## 5. Verify the connector 1. Confirm `GET /.status` succeeds on the public backend origin. 2. Confirm the worker is running and shares the backend encryption key. 3. Open **Connectors**, select **Add connection**, then select **Connect** on the **Linear** row. 4. If `LINEAR_*` is unset, complete **Register Linear OAuth app**, then Connect. 5. Complete OAuth and confirm the expected Linear workspace appears. 6. In **Add to Workspaces**, choose a Workspace and a scope. 7. Merge the pull request that adds `sources/linear/.md`. 8. Confirm the GitHub App delivers the merge push to `{AUTH_BASE_URL}/api/v1/webhook/github/`. 9. Confirm the connector reaches **Connected** and generated files appear under `linear/` in the Workspace repository. 10. Edit a mirrored Linear issue and confirm `POST /api/v1/webhook/linear` succeeds and the corresponding Git file changes. 11. Confirm the Workspace re-indexes. ctxpipe copies Linear file uploads and explicit embedded external media into sibling `assets/` directories in the Workspace repository. Ordinary link-only attachments stay links. ctxpipe never crawls them. GitHub pull requests and commits stay references. The limits are 25 MiB for each asset and 100 MiB for each Linear entity. An unsafe, failed, or oversized asset leaves a Linear link or a textual stub. Give the destination repository the same sensitivity as the selected Linear scope. Initial sync proves OAuth, worker, and GitHub access. The webhook step separately proves Linear webhook delivery and signature configuration. ## Troubleshooting ### “Linear OAuth is not configured” Neither the connection nor the deployment has a Linear client ID and secret. Register the app in the wizard, or set `LINEAR_CLIENT_ID` and `LINEAR_CLIENT_SECRET` on the backend and the worker if you want a deployment-wide app. ### Linear reports a redirect URI mismatch Compare the authorize request's `redirect_uri` with the application setting character for character. If you can, leave `LINEAR_REDIRECT_URI` unset, so that ctxpipe derives it from the public `AUTH_BASE_URL`. Use the override only behind a proxy with a different external callback. ### OAuth succeeds but sync later fails The worker must decrypt the saved client secret. It needs the same `AUTH_SECRET` / `CONNECTION_SECRETS_ENCRYPTION_KEY` as the backend. When you use the environment variable fallback, the worker needs `LINEAR_CLIENT_ID` and `LINEAR_CLIENT_SECRET`. Check outbound HTTPS and the worker logs. Check if someone revoked the application authorization in Linear. ### The source pull request cannot open If **Add to Workspaces** reports an error, confirm that the selected GitHub App installation still has write access to the repository of the Workspace. Also confirm that the Workspace repository uses format 2 or higher. Below that, the dialog shows **Needs the format upgrade first.** next to the Workspace. ### Webhook endpoint returns 503 No webhook signing secret is present on the matching Linear connection or in `LINEAR_WEBHOOK_SECRET`. ### Webhook endpoint returns 401 The signing secret does not match the Linear application that sent the event, a proxy changed the raw body, or the event timestamp is stale. Do not parse and re-serialize the body in a gateway before forwarding it. A secret saved on one connection does not authorize events for another. ### Initial sync works but later edits do not appear This is a webhook problem, not an OAuth problem. Confirm that webhooks are enabled on the application, the public URL is reachable, the relevant resource categories are selected, and requests reach `/api/v1/webhook/linear`. ## User guide See [Linear connector](/docs/connections/source-connectors/linear) for scope, repository layout, review workflow, and user-facing recovery steps. # Notion (https://docs.ctxpipe.ai/docs/self-hosting/notion) ## Overview Self-hosted Notion uses the same connector routes as hosted ctxpipe. The main path is to register a **private Notion OAuth app** in the product UI. Then you can connect a workspace without `NOTION_*` environment variables and without a restart. Environment variables stay as an optional shortcut for hosted and operator setups. Traffic stays on **this** deployment's `AUTH_BASE_URL`. Do not point OAuth callbacks or webhook URLs at hosted ctxpipe. ## 1. Register the OAuth app in the UI 1. Open **Connectors**, select **Add connection**, then select **Connect** on the **Notion** row. ctxpipe creates a draft Notion connection. 2. On **Register Notion OAuth app**, create an OAuth app in the [Notion integrations portal](https://www.notion.so/profile/integrations). For a private install, choose **Selected workspaces only**. Do not list the app on the Marketplace. An internal (token-only) integration does not work. 3. Copy the callback URL from the wizard into the app's OAuth / redirect URI field. It is: ```text {AUTH_BASE_URL}/api/v1/connectors/notion/oauth/callback ``` Set **`NOTION_REDIRECT_URI`** only when the public callback differs from the URL that ctxpipe derives from `AUTH_BASE_URL`. 4. Paste the **Client ID** and **Client secret**, then save. ctxpipe stores the secret encrypted on that connection. It never shows the secret again. 5. On **Add the webhook**, open the **Webhooks** tab of the same app. Add a webhook. Paste the wizard event URL into **Webhook URL**. Select the page, data source, and database event types (`page.*`, `data_source.*`, and `database.*`). ctxpipe ignores other events. If you do not select a type, ctxpipe does not get its updates. Query parameters scope the URL to this connection: ```text {AUTH_BASE_URL}/api/v1/webhook/notion?connectionId={id}&provisioningToken={hmac} ``` The provisioning token is an HMAC of the saved client secret. Notion's first verification request is unsigned. ctxpipe checks that HMAC. It stores the verification token on the same row and shows it in the wizard. Paste that **Verification token** into Notion's **Verify subscription** dialog. Then select **Verify subscription**. 6. Choose **Continue**, then **Connect Notion**. Then authorize the workspace. The worker reads the OAuth client credentials from the connection row. Token refresh therefore works when the `NOTION_*` environment variables are empty. ## 2. Optional env shortcut Operators who have a shared OAuth app can set **`NOTION_CLIENT_ID`**, **`NOTION_CLIENT_SECRET`**, and **`NOTION_WEBHOOK_SECRET`** on the backend and worker instead of saving them in the wizard. When you set those environment variables, **Add Notion** skips the register step and opens the OAuth popup. For webhook provisioning with environment variables only (no `connectionId` on the webhook URL), generate the provisioning token from the repository root: ```bash NOTION_CLIENT_SECRET='your-client-secret' bun -e 'import { createHmac } from "node:crypto"; process.stdout.write(createHmac("sha256", process.env.NOTION_CLIENT_SECRET ?? "").update("ctxpipe:notion-webhook-provisioning:v1").digest("base64url"))' ``` Use the result in: ```text {AUTH_BASE_URL}/api/v1/webhook/notion?provisioningToken={generated-token} ``` Copy the **verification token** from Notion after the first delivery. Set **`NOTION_WEBHOOK_SECRET`** to it. Provisioning with environment variables only does not save that token on a connection row. Do not ask each ctxpipe organization to create a second webhook path. One webhook URL for each integration is enough. A self-hosted deployment scopes it with query parameters. ## 3. Supply env secrets when you use the shortcut ### Docker Compose Add the values to the root `.env`, then restart the backend and the worker: ```bash NOTION_CLIENT_ID=... NOTION_CLIENT_SECRET=... NOTION_WEBHOOK_SECRET=... pnpm start ``` ### AWS CDK Pass `notionClientId`, `notionClientSecret`, and `notionWebhookSecret` through the construct's `connectorSecrets`. Keep the values in your CDK secret source. Do not commit them in stack code. ## 4. Complete connector setup After you register the integration in the UI or with environment variables: 1. Authorize the Notion workspace. 2. In **Add to Workspaces**, select the Workspaces that receive Notion content. 3. Select the Notion pages and databases to mirror. 4. Review and merge the pull request that adds `sources/notion/.md`. ctxpipe writes pages as Markdown under `notion/pages/`. It writes databases under `notion/databases/`. Each database has one `table.csv` snapshot, and each row has one Markdown file, for task and knowledge retrieval. ctxpipe copies provider-declared files and explicit embedded external media into an `assets/` directory beside the page or row that uses them. The limits are 25 MiB for each file and 100 MiB for each page or row. ctxpipe never saves Notion URLs that expire. A failed, unsafe, or oversized download leaves a Notion page link or a caption stub. ## Verify the complete flow 1. Add Notion and save a private OAuth app (or confirm that the environment variables are set). 2. Authorize a clean Notion workspace through the product flow. 3. Share a page or database with the integration and confirm it appears. 4. Select a Workspace in **Add to Workspaces** and merge the pull request that adds `sources/notion/.md`. 5. Confirm the initial mirror writes readable files under `notion/`. 6. Edit an in-scope source and confirm the deployment webhook receives the event. 7. Confirm the generated git file changes and the Workspace re-indexes. ## Troubleshooting ### OAuth returns to the wrong deployment The integration callback must match the public `AUTH_BASE_URL` exactly, including scheme, host, and path. Set `NOTION_REDIRECT_URI` only when the external route differs. ### Register step does not appear The wizard skips registration when both `NOTION_CLIENT_ID` and `NOTION_CLIENT_SECRET` are set. That is the hosted/operator shortcut. ### Signed events return `401` or `503` If you used the UI webhook URL, confirm Notion's verification completed on that connection (the webhook URL includes `connectionId`). If you used the environment variable shortcut, set `NOTION_WEBHOOK_SECRET` to the verification token shown after Notion's first delivery, then restart the backend. The provisioning token in the webhook URL does not replace the signing secret. ### Content sync cannot refresh authorization Confirm the connection still has an OAuth app (or that both backend and worker receive the same Notion client ID and secret) and can reach `api.notion.com`. ### No pages appear Notion returns only the pages and databases that you share with the integration. Share the resource in Notion. Then refresh resources in the ctxpipe setup flow. ## User guide See [Notion connector](/docs/connections/source-connectors/notion) for scope, repository layout, review workflow, and user-facing recovery steps. ## Related pages * [Workspaces](/docs/workspaces) * [GitHub connector](/docs/connections/source-connectors/github) * [Indexing](/docs/workspaces/indexing) # PagerDuty (https://docs.ctxpipe.ai/docs/self-hosting/pagerduty) ## Overview Self-hosted PagerDuty uses the same connector flow as hosted ctxpipe. Someone must still create a Scoped OAuth application in the [PagerDuty developer portal](https://developer.pagerduty.com). The usual path is to paste the client ID and secret of that app in the connector wizard. ctxpipe then creates one v3 webhook subscription for each connection. Operators do not paste a shared webhook signing secret. Deployment environment variables (`PAGERDUTY_CLIENT_ID` / `PAGERDUTY_CLIENT_SECRET`) are an optional shortcut, as on hosted ctxpipe. You do not need to restart Compose or CDK to finish first-time setup. ## What you need * A public HTTPS `AUTH_BASE_URL`. In production, keep `CTXPIPE_PUBLIC_APP_URL` on the same origin. * Permission to create a PagerDuty OAuth application. * A ctxpipe GitHub App connection with write access to each Workspace repository that syncs PagerDuty incidents. * Network access from the backend and the worker to `identity.pagerduty.com` and `api.pagerduty.com` or `api.eu.pagerduty.com` over HTTPS. PagerDuty cannot deliver webhooks to a private localhost address. To test a connector end to end, use a public HTTPS deployment or tunnel. ## 1. Register the OAuth app in the wizard 1. Open **Connectors**, select **Add connection**, then select **Connect** on the **PagerDuty** row. ctxpipe creates a draft connection so that you can save the app before account authorization. 2. Open [PagerDuty App Registration](https://developer.pagerduty.com/my-apps), select **New App**. Name the app `ctxpipe` or use the name of your deployment. 3. Under **Functionality**, select **OAuth 2.0**. Leave **Events Integration** unselected. ctxpipe creates a v3 webhook subscription through PagerDuty's API after account authorization. This is not an Events API integration. 4. Choose **Scoped OAuth**. Register this redirect URL exactly. The wizard copies it for you: ```text {AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback ``` 5. Grant only these permission scopes: * `incidents.read` * `services.read` * `users.read` * `webhook_subscriptions.read` * `webhook_subscriptions.write` 6. Select **Register App**. Copy the generated client id and client secret before you close the confirmation. PagerDuty shows the secret only once. 7. Save both values in **Register PagerDuty OAuth app**. ctxpipe encrypts the secret on the connection. It never returns the secret from connector metadata. Continue to **Connect PagerDuty**. Do not add write-back scopes for incidents. ctxpipe does not acknowledge or resolve PagerDuty incidents. ctxpipe creates webhook signing secrets for each subscription and stores them encrypted on the connection. Do not create a deployment-wide PagerDuty webhook secret. ## 2. Optional: deployment env shortcut If you want one shared app for the whole deployment, set **`PAGERDUTY_CLIENT_ID`** and **`PAGERDUTY_CLIENT_SECRET`** on the backend and the worker. Set **`PAGERDUTY_REDIRECT_URI`** only when the public callback differs from the URL that ctxpipe derives from `AUTH_BASE_URL`. The wizard then starts at **Connect PagerDuty**. ctxpipe looks for credentials in the saved connection app first. Then it looks in the environment variables. The worker needs the client secret that the refresh path reads. ### Docker Compose Add the values to the root `.env`, then restart the backend and the worker: ```bash PAGERDUTY_CLIENT_ID=... PAGERDUTY_CLIENT_SECRET=... pnpm start ``` ### AWS CDK Pass the values through `connectorSecrets`: ```ts new CtxPipe(stack, "CtxPipe", { // other required settings connectorSecrets: { pagerdutyClientId: cdk.SecretValue.secretsManager("pagerduty/client-id"), pagerdutyClientSecret: cdk.SecretValue.secretsManager( "pagerduty/client-secret", ), // Optional when AUTH_BASE_URL already produces the registered callback: pagerdutyRedirectUri: cdk.SecretValue.unsafePlainText( "https://app.example.com/api/v1/integrations/pagerduty/callback", ), }, }); ``` The construct stores these values in the connector secret and injects them into backend and worker task definitions. ## 3. Public webhook URL After a user completes OAuth, ctxpipe creates one account-level v3 webhook subscription for incident events and points it at: ```text {AUTH_BASE_URL}/api/v1/webhook/pagerduty ``` Do not paste this URL into **Events Integration** during App Registration. PagerDuty receives it when ctxpipe creates the webhook subscription through the Webhook Subscriptions API. ctxpipe verifies `X-PagerDuty-Signature` HMAC-SHA256 against the raw request body. It binds the event to the one connection whose stored secret verifies. It routes with `X-PagerDuty-Subscription`. Re-authorization replaces a lost secret instead of stacking another subscription on the same event URL. Account-level delivery can include every service. ctxpipe syncs an event only for the Workspace sources whose services include the incident's service. ctxpipe acknowledges events with no service, or with a service that no source selects, with `200`. It never fetches them. ## 4. Verify the connector 1. Confirm `GET /.status` succeeds on the public backend origin. 2. Open **Connectors**, select **Add connection**, then select **Connect** on the **PagerDuty** row. 3. If the wizard shows **Register PagerDuty OAuth app**, save the client id and secret. Skip this when the deployment already has a shared app that environment variables set. 4. Complete OAuth and confirm the expected account appears. 5. In **Add PagerDuty to Workspaces**, select one or more Workspaces and at least one service for each. Use search. Show more services if the account has many. 6. Merge the pull request that adds `sources/pagerduty/.md` in each Workspace repository. 7. Confirm the source shows **Connected** in the Workspace settings and generated files appear under `pagerduty/services//incidents/`. 8. Trigger or update an in-scope incident and confirm `POST /api/v1/webhook/pagerduty` succeeds and the corresponding Git file changes. 9. Confirm the Workspace re-indexes. ## Upgrade from `pagerduty/config.yaml` Earlier versions kept the service scope in `pagerduty/config.yaml` in one sync repository. Upgrade step 1 of each Workspace moves the connection to Workspace sources. An org owner or admin starts it with **Start** in the Upgrades section of the Workspace settings. You can also start it from the command line: ```bash pnpm --filter @ctxpipe/backend migrate-workspace-sources --org pnpm --filter @ctxpipe/backend migrate-workspace-sources --org --apply ``` The first command only prints the plan. With `--apply`, the command starts the Workspace upgrade, as the button does. The upgrade opens one pull request for each Workspace and never pushes to the default branch. When the Workspace links the old sync repository, the pull request adds the merged services as a source. The old per-connection sync stops after the merge, when the new source syncs. When the Workspace repository is the old sync repository, the same pull request moves each file in `pagerduty/incidents/` to `pagerduty/services//incidents/`, with its assets. The service id comes from the `service_id` front matter of the file. Git keeps the history of each moved file. A file without a readable `service_id` stays in `pagerduty/incidents/`. ## US and EU accounts ctxpipe finds the account region from the authorized user's `html_url` after OAuth. Do not add a region field to the wizard. The API host is `api.pagerduty.com` or `api.eu.pagerduty.com`. ## Troubleshooting ### “PagerDuty OAuth is not configured” Neither the connection nor the deployment has OAuth client credentials. Finish **Register PagerDuty OAuth app** in the wizard, or set both `PAGERDUTY_CLIENT_ID` and `PAGERDUTY_CLIENT_SECRET` on the backend and the worker and restart. The catalog still opens the setup wizard. Connect fails when ctxpipe cannot create the authorization URL. ### Callback is rejected The registered redirect URI must match `{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback` or `PAGERDUTY_REDIRECT_URI` exactly, including scheme, host, path, and whether a trailing slash is present. ### Incremental events never arrive Confirm these three points: the public webhook URL is reachable, the connection is connected, and the service of the incident is in a **Connected** PagerDuty source of a Workspace. ctxpipe rejects tampered or stale signatures. A failed enqueue returns `5xx`, so PagerDuty can retry. ### A source shows “Out of access” Someone deleted a selected service in PagerDuty, or the OAuth grant cannot read it. ctxpipe tries the service in three sync cycles and keeps its files. Then it marks only that source. Remove the service from the source, or give the connection access to it. See [PagerDuty connector](/docs/connections/source-connectors/pagerduty) for scope, files, and product troubleshooting. # Plain (https://docs.ctxpipe.ai/docs/self-hosting/plain) ## Overview Self-hosted Plain uses the same connector flow as hosted ctxpipe. Plain has no OAuth application for other products. An organization admin pastes the API key of a Plain machine user in the setup. ctxpipe encrypts the key and the webhook token on the connection with the connection secret key of the deployment. There is no `PLAIN_*` environment variable. You do not need to change Compose or CDK settings. ## What you need * A public HTTPS `AUTH_BASE_URL` for the webhook target. Plain sends webhooks only to a public HTTPS address. Without it, the connector syncs once a day and on **Sync now**. ctxpipe cannot check that Plain reaches the URL. When Plain accepts the target but cannot reach it, the card still shows live updates as on. * A ctxpipe GitHub App connection with write access to each Workspace repository that syncs Plain content. * Network access from the backend and the worker to `core-api.uk.plain.com` over HTTPS. ## Webhook target At connect, ctxpipe creates one Plain webhook target for each connection: ```text {AUTH_BASE_URL}/api/v1/webhook/plain/{connectionId} ``` The target sends `timeline.timeline_entry_changed` events with a random token in the `X-Ctxpipe-Webhook-Token` header. ctxpipe compares the token and refuses a request without it. A failed enqueue returns `503`. Plain then sends the event again. ## Verify the connector 1. Open **Connectors**, select **Add connection**, then select **Connect** on the **Plain** row. 2. Paste the API key of a machine user with the permissions that the setup lists. 3. In the **Add to Workspaces** step, select a Workspace and at least one Help Center or label. Merge the pull request that adds the source file under `sources/plain/`. 4. Confirm the source shows **Connected** in the Workspace settings and files appear under `plain/`. 5. In Plain, open **Settings → Webhooks** and confirm the ctxpipe target. Add a note to a real thread with a selected label. Do not use a Plain test thread. Confirm that `POST /api/v1/webhook/plain/{connectionId}` returns `200`. Confirm that the thread file changes after the 10-minute window. See [Plain connector](/docs/connections/source-connectors/plain) for the files, the sync, and product troubleshooting. # Slack (https://docs.ctxpipe.ai/docs/self-hosting/slack) ## Overview Self-hosted Slack uses the same connector flow as hosted ctxpipe. But your deployment owns **one** Slack app. Organizations authorize that app into their workspace. They do not create their own Slack apps or webhook URLs. This page covers only the operator setup. It has the Slack app, its scopes and events, and the secrets. For how people capture threads, see the [Slack connector guide](/docs/connections/source-connectors/slack). ## What you need * A public HTTPS `AUTH_BASE_URL` that Slack can reach. * A Slack app with OAuth and Event Subscriptions enabled. * A GitHub connection, and a Workspace that the GitHub App can write to. * Backend environment variables: `SLACK_CLIENT_ID`, `SLACK_CLIENT_SECRET`, `SLACK_SIGNING_SECRET`. * Model environment variables (`MODEL_PROVIDER*` / Bedrock), so the mention agent can read extra text after `@mention`. A bare mention captures without the model. * The same stable `AUTH_SECRET` and, when set, `CONNECTION_SECRETS_ENCRYPTION_KEY` on backend and worker so the worker can decrypt stored bot tokens. ## 1. Create the Slack app In the [Slack API dashboard](https://api.slack.com/apps), create an app. You can start from scratch. For higher rate limits in production, distribute it as a Marketplace app or an internal organization app. Do not use a throwaway development app. If organizations will install the app into workspaces other than the app owner's development workspace, activate **Manage Distribution** in Slack. Marketplace listing is a separate step. Private distribution to many workspaces does not need it. ### OAuth & Permissions Add at least these bot token scopes: * `app_mentions:read` (receive capture requests from bot mentions) * `channels:history`, `channels:read` * `groups:history`, `groups:read` (private channels; invite the bot) * `chat:write` (in-thread capturing → captured status replies) * `files:read` * `users:read` If you add a scope after a workspace installs the app, that workspace must complete OAuth again. A change to the Slack app settings does not upgrade the bot token that you have. Register this OAuth redirect URL: ```text {AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback ``` Set **`SLACK_CLIENT_ID`** and **`SLACK_CLIENT_SECRET`** on the backend from the app credentials. Set **`SLACK_REDIRECT_URI`** only when the public callback differs from the URL derived from `AUTH_BASE_URL`. ### Event Subscriptions Enable Event Subscriptions and set the Request URL to: ```text {AUTH_BASE_URL}/api/v1/webhook/slack ``` Slack verifies the URL with a signed `url_verification` challenge. ctxpipe checks `X-Slack-Signature` using **`SLACK_SIGNING_SECRET`**. Subscribe the bot to: * **`app_mention`**: the capture requests; * **`app_uninstalled`** and **`tokens_revoked`**: ctxpipe marks the connection revoked when a workspace removes the app. Do **not** subscribe to `message.channels` / `message.groups` for the connector ingest path. Continuous message fan-out is not used. **Turn Socket Mode off.** If Socket Mode is enabled, Slack does not POST to the Request URL (“You won’t need a Request URL”) and mentions never reach ctxpipe. The Events URL is deployment-wide. Do not ask each organization to register its own webhook. ## 2. Wire deployment secrets Set all three Slack values together. A partial configuration leaves OAuth or event verification unusable. ### Docker Compose Add the values to the root `.env` used by the `deploy` profile, then recreate the backend and the worker: ```dotenv SLACK_CLIENT_ID=... SLACK_CLIENT_SECRET=... SLACK_SIGNING_SECRET=... ``` ```bash docker compose --profile deploy up -d --force-recreate backend worker ``` ### AWS CDK Pass Slack credentials through `connectorSecrets`: ```ts new CtxPipe(stack, "CtxPipe", { // ...required props connectorSecrets: { slackClientId: cdk.SecretValue.secretsManager("slack-client-id"), slackClientSecret: cdk.SecretValue.secretsManager("slack-client-secret"), slackSigningSecret: cdk.SecretValue.secretsManager("slack-signing-secret"), }, }) ``` Use your normal secret-management convention. Do not put plaintext credentials in source control. ## 3. Check the connector After the Slack app is configured, an organization admin connects Slack on **Connectors**, adds the channels to a Workspace, and merges the source pull request. See the [Slack connector guide](/docs/connections/source-connectors/slack). * One Slack workspace connects to one ctxpipe organization. * When you reinstall or rotate the Slack app, the bot leaves its channels. Invite it again in each channel. * All organizations of the deployment share the rate limits of the one Slack app. ## Operator troubleshooting ### “Slack is not configured on this deployment” The backend does not have both OAuth values. Confirm that `SLACK_CLIENT_ID` and `SLACK_CLIENT_SECRET` are not empty in the backend runtime. Then redeploy or restart the backend. A change to repository secrets or CI secrets alone does not update a running service. ### OAuth returns `invalid_redirect_uri` Register the exact callback: ```text {AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback ``` Set `SLACK_REDIRECT_URI` only when you use a different public callback on purpose. Its value must exactly match a redirect URL registered in Slack. ### Slack cannot verify the Events Request URL * Confirm the public route is `{AUTH_BASE_URL}/api/v1/webhook/slack`. * Confirm `SLACK_SIGNING_SECRET` is Slack's **Signing Secret**, not the legacy verification token. * Confirm the public proxy forwards the raw request body and `X-Slack-Signature` / `X-Slack-Request-Timestamp` headers unchanged. * An unsigned test POST must return 401. A 404 means that the routing is wrong. A 503 means that the backend has no signing secret. ### Mentions do not create commits * Confirm Socket Mode is **off** and the bot Event Subscription includes **`app_mention`**. * Confirm that a Slack source with the status **Connected** lists the channel. If no Workspace has the channel, the bot replies "Nothing captured: no Workspace has this channel." * Confirm the bot is a member of the channel. After an app reinstall, invite it again. * Mention the bot's **@handle** (Slack autocomplete), not only a display name typed as plain text. * Read the worker logs for the Slack mention capture job. The backend and the worker must use the same database and stable encryption material (`AUTH_SECRET` and, when set, `CONNECTION_SECRETS_ENCRYPTION_KEY`). The GitHub App must still have contents write access to the Workspace repository. Extra text after a mention needs `MODEL_PROVIDER*` (or Bedrock). A second organization cannot install a workspace through OAuth when that workspace is already connected. ### Capture commits but the repository never indexes Capture writes to git through the GitHub App. Indexing starts only when this deployment receives the GitHub **push** webhook for that repository. Confirm that the GitHub App webhook URL points at **this** origin (`{AUTH_BASE_URL}/api/v1/webhook/github/...`) and not at another environment. Confirm that the worker and codesearch services run. ## Related pages * [Configuration](/docs/self-hosting/configuration) * [Slack connector (product)](/docs/connections/source-connectors/slack) * [Confluence & Atlassian](/docs/self-hosting/confluence-atlassian) * [GitHub App connector](/docs/self-hosting/configuration#github-app-connector) * [Slack MCP server (external)](https://docs.slack.dev/ai/slack-mcp-server) # AWS (https://docs.ctxpipe.ai/docs/self-hosting/deployment/aws) `@ctxpipe/aws-cdk` is the AWS self-hosting path for ctxpipe. It gives you one high-level construct, `CtxPipe`. The construct provisions the required infrastructure and sets the runtime configuration for you. Self-hosted ctxpipe on `@ctxpipe/aws-cdk` is **single-tenant**. Run one stack per organization. ## Step-by-step deployment flow ### 1) Install `@ctxpipe/aws-cdk` in your CDK repo Use an existing CDK repo, or create a new one first. Then install: ```bash pnpm add @ctxpipe/aws-cdk aws-cdk-lib constructs ``` ### 2) Create or reuse a Route 53 hosted zone You need a hosted zone for your custom domain because the construct configures DNS and TLS from it. * Reuse an existing public hosted zone if you already manage your domain in Route 53. * Or create a new hosted zone and delegate your domain or subdomain to it. * Keep both values ready: * `customDomain.domainName` (example: `app.example.com`) * `customDomain.hostedZoneId` (example: `Z0123456789ABCDEF`) ### 3) Import `CtxPipe` and configure it Add the construct to your stack. Follow the same pattern as [`examples/aws-cdk-self-host`](https://github.com/ctxpipe-ai/ctxpipe/tree/main/examples/aws-cdk-self-host). `modelProvider` is a union of two variants: **openai-like** (default) and **bedrock**. Both variants set the tiers through the same `models` prop (`fast` is required). #### OpenAI-compatible provider ```ts import * as cdk from "aws-cdk-lib"; import { CtxPipe } from "@ctxpipe/aws-cdk"; const app = new cdk.App(); const stack = new cdk.Stack(app, "CtxPipeStack"); const modelApiKey = app.node.tryGetContext("modelApiKey"); if (!modelApiKey) { throw new Error('Missing CDK context "modelApiKey". Pass -c modelApiKey=...'); } new CtxPipe(stack, "CtxPipe", { orgSlug: "acme", size: "medium", customDomain: { domainName: "app.example.com", hostedZoneId: "Z0123456789ABCDEF", }, modelProvider: { baseUrl: "https://api.openai.com/v1", apiKey: cdk.SecretValue.unsafePlainText(String(modelApiKey)), models: { fast: "gpt-4.1-mini", embedding: "text-embedding-3-large", // Optional: medium, high // medium: "gpt-4.1", // high: "gpt-4.1", }, }, }); ``` Set `models.embedding` for an OpenAI-compatible provider. Without it, the backend uses `openai/text-embedding-3-large`, an OpenRouter model id that other providers do not know. #### Amazon Bedrock (task role) For Bedrock, set `kind: "bedrock"` and the tier model IDs only. The construct grants the backend and worker ECS task roles `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`. It injects `MODEL_PROVIDER=bedrock` and `MODEL_BEDROCK_AWS_REGION`. It does **not** create a `MODEL_PROVIDER_API_KEY` secret. The backend calls Bedrock Runtime with SigV4 credentials from the ECS task role. Enable each model ID in the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/) for your stack region before you use chat or embeddings. Bedrock embeddings currently support only **Cohere** embed models. The backend uses the Cohere invoke format. When you omit `models.embedding`, the default is `cohere.embed-v4:0`. You can also use inference-profile IDs such as `global.cohere.embed-v4:0`. Non-Cohere Bedrock embed models (for example Titan) are not supported yet. The construct attaches `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` to the backend and worker task roles. An AWS Organizations **service control policy (SCP)** can override them with an **explicit deny**. The log text is typically `AccessDeniedException` … `explicit deny in a service control policy`. Allow your chat and Cohere embed foundation-model ARNs in the SCP. Denies for `global.cohere.embed-v4:0` often name `…/foundation-model/cohere.embed-v4:0`. This is a different problem from chat stream stalls such as `Bedrock Converse stream timed out … without receiving a chunk`. For those stalls, investigate `ConverseStream` in CloudTrail. A wider task-role policy does not help. ```ts new CtxPipe(stack, "CtxPipe", { orgSlug: "acme", customDomain: { domainName: "app.example.com", hostedZoneId: "Z0123456789ABCDEF", }, modelProvider: { kind: "bedrock", region: "us-east-1", // optional; defaults to stack region models: { fast: "anthropic.claude-sonnet-4-20250514-v1:0", medium: "anthropic.claude-sonnet-4-20250514-v1:0", high: "anthropic.claude-opus-4-20250514-v1:0", embedding: "cohere.embed-v4:0", // optional; Cohere models only (e.g. global.cohere.embed-v4:0) }, }, }); ``` ### 4) Run CDK deploy **OpenAI-like:** ```bash cdk synth cdk deploy -c modelApiKey="$OPENAI_API_KEY" ``` For **Bedrock**, use the example app in [`examples/aws-cdk-self-host`](https://github.com/ctxpipe-ai/ctxpipe/tree/main/examples/aws-cdk-self-host). It always uses Bedrock. It reads `orgSlug`, `domainName`, `hostedZoneId`, `size`, `stackName`, `modelFast`, `modelMedium`, `modelHigh`, and `modelEmbedding` from the context: ```bash cdk deploy \ -c modelFast="anthropic.claude-sonnet-4-20250514-v1:0" \ -c orgSlug=acme \ -c domainName=app.example.com \ -c hostedZoneId=Z0123456789ABCDEF ``` ### 5) Wait for deployment and open your custom domain After `cdk deploy` completes, open `https://`. ### 6) Complete onboarding with the same `orgSlug` When you create the first organization in onboarding, use the **same** slug as the `orgSlug` of `CtxPipe`. The construct uses the slug for the graph database settings of the organization (`GRAPH_DB_URI_`). If the two slugs are different, the graph of the organization does not work. The slug can contain only lowercase letters, digits, and `-`. ### 7) Upgrade an existing stack A new `@ctxpipe/aws-cdk` version on npm is the release unit. It pins the backend, worker, UI, codesearch, and migrate images to one git SHA. It also includes the Fargate size profiles of that release. After npm publishes the version: ```bash pnpm update @ctxpipe/aws-cdk cdk synth cdk deploy ``` Before the first deploy of a new version, check the Aurora minor version. See [Aurora PostgreSQL minor version](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#aurora-postgresql-minor-version). `cdk deploy` runs Postgres migrations. Then it rolls the ECS services. Do not set image tags in `CtxPipe` props. Do not retag tasks to `:latest` yourself. The migrate task provisions `ctxpipe_app`. Then the stack writes the runtime `DATABASE_URL` secret. CDK operators do not set either URL by hand and do not run `psql`. This way, the codesearch memory changes (4 / 8 / 12 GiB) and the ingest behavior arrive together. An update of only the container images, or only the task memory, leaves the stack split across releases. The first upgrade to a version with the [chat sandbox host](#chat-sandbox-host) adds it with no new required props. That deploy only adds resources and replaces nothing that holds data. It waits until the host reports ready. Then it rolls the backend and the worker. The first upgrade to a version with the [knowledge store](#knowledge-store) also adds it with no new required props, and replaces nothing that holds data. The deploy waits until ClickHouse answers. Then the migrate task makes the ClickHouse tables and users, and the deploy rolls the services. To automate upgrades, let Renovate or Dependabot open a pull request for each new `@ctxpipe/aws-cdk` version, and run `cdk synth` and `cdk deploy` from your pipeline. ## Why each prop exists ### Required props | Prop | Why it is required | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `orgSlug` | Self-hosted AWS CDK deployments are single-tenant. The slug identifies the one org for this stack and is used for org-scoped graph runtime values such as `GRAPH_DB_URI_`. | | `customDomain` | ctxpipe needs one stable public HTTPS origin for browser traffic, auth, API routes, connector callbacks, and MCP. The construct uses this to configure DNS + TLS + ALB listeners. | | `modelProvider` | Backend and worker need model IDs and either an OpenAI-compatible endpoint + API key, or Bedrock via ECS task role. See union below. | ### Optional props | Prop | Behavior | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `size` | Capacity profile for single-tenant deployments. Valid values: `small`, `medium`, `large`. Defaults to `small` when omitted. | | `otel` | Optional OTLP export. Omit it and the tasks export nothing. The construct does not deploy a collector or ClickStack. | | `sandboxHost` | The [chat sandbox host](#chat-sandbox-host): `instanceType`, `dockerVolumeSizeGiB` (a whole number, at least 20), `agentVaultNetworkAllowlist` for a [private git server](#private-git-servers), and `enabled`. The host is on by default. `enabled: false` creates no host, and turns chat off. | | `connectorSecrets` | Deployment-wide connector credentials. See [Optional connector secrets](#optional-connector-secrets). | ### `modelProvider` union | Variant | `kind` | Required nested fields | Runtime auth | | ----------------- | ------------------------ | ---------------------------------- | ----------------------------------------------------------------------- | | OpenAI-compatible | omitted or `openai-like` | `baseUrl`, `apiKey`, `models.fast` | `MODEL_PROVIDER_API_KEY` in Secrets Manager | | Amazon Bedrock | `bedrock` | `models.fast` | ECS task role → native Bedrock Runtime (SigV4); no model API key secret | Optional on both variants: `models.medium`, `models.high`, and `models.embedding`. Omitted tier IDs cascade: **medium** → **fast**, **high** → **medium**. The construct maps them to `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, `MODEL_HIGH_NAME`, and `MODEL_EMBEDDING_NAME`. See [Model configuration](/docs/self-hosting/models) for what each tier is used for. For **bedrock**, `modelProviderSecret` / `modelProviderSecretArn` stack outputs are omitted. ### Upgrading from older `@ctxpipe/aws-cdk` versions | Removed | Replacement | | ---------------------------- | -------------------------------------- | | `modelProvider.defaultModel` | `modelProvider.models.fast` (required) | Replace `defaultModel: "gpt-4.1-mini"` with `models: { fast: "gpt-4.1-mini" }`. ### Required nested fields (openai-like) | Field | Why it is required | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `customDomain.domainName` | Becomes the public app origin (`AUTH_BASE_URL` / `CTXPIPE_PUBLIC_APP_URL`) served over HTTPS. This is the URL users visit for sign-in and onboarding. | | `customDomain.hostedZoneId` | Used to create/validate Route 53 records for ACM certificate validation, ALB alias records, and SES domain identity/DKIM records. | | `modelProvider.baseUrl` | Endpoint for OpenAI-compatible model APIs used by backend and worker. | | `modelProvider.apiKey` | Secret used to authenticate model API requests at runtime. | | `modelProvider.models.fast` | Model ID for the fast tier (`MODEL_FAST_NAME`). | ### Required nested fields (bedrock) | Field | Why it is required | | --------------------------- | -------------------------------------------------------------------------------------------------------------- | | `customDomain.domainName` | Same as openai-like — public HTTPS origin. | | `customDomain.hostedZoneId` | Same as openai-like — DNS/TLS/SES records. | | `modelProvider.models.fast` | Bedrock model ID for the fast tier (`MODEL_FAST_NAME`). Enable it in the Bedrock console for the stack region. | ### Optional connector secrets Pass deployment-wide connector credentials through `connectorSecrets`. Each value is a `cdk.SecretValue`. The construct stores them in Secrets Manager and gives them to the ECS tasks that need them. Most connectors can also take their credentials in the setup wizard of the product, so these are optional. | Connector | Keys | | ---------- | -------------------------------------------------------------------------------------------------------- | | GitHub App | `githubAppId`, `githubPrivateKey`, `githubWebhookSecret`, `githubClientId`, `githubClientSecret` | | Atlassian | `atlassianClientId`, `atlassianClientSecret` | | Slack | `slackClientId`, `slackClientSecret`, `slackSigningSecret` (set all three) | | Linear | `linearClientId`, `linearClientSecret`, `linearRedirectUri`, `linearWebhookSecret` | | Notion | `notionClientId`, `notionClientSecret`, `notionWebhookSecret` | | PagerDuty | `pagerdutyClientId`, `pagerdutyClientSecret`, `pagerdutyRedirectUri`. There is no shared webhook secret. | See [Self-host Slack](/docs/self-hosting/slack) and [Self-host PagerDuty](/docs/self-hosting/pagerduty) for callback and webhook settings. ## Observability | Prop | Env on backend, worker, UI, and codesearch | | -------------------- | --------------------------------------------------- | | `endpoint` | `OTEL_EXPORTER_OTLP_{TRACES,LOGS,METRICS}_ENDPOINT` | | `headers` | `OTEL_EXPORTER_OTLP_HEADERS` (Secrets Manager) | | `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` | [Configuration](/docs/self-hosting/configuration) describes browser telemetry and the signal URLs that the apps read. For the prop reference, see the [package README](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#observability). ## High-level architecture `CtxPipe` provisions a single AWS stack with these layers: | Layer | What is provisioned | Purpose | | ------------------ | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | | Network | VPC with public/private subnets, NAT egress, and an S3 gateway endpoint | Isolates internal services while allowing controlled outbound access. S3 traffic does not go through the NAT gateway. | | Edge + ingress | Public ALB + ACM certificate + Route 53 records | Exposes one HTTPS origin and routes public traffic to backend. | | Compute | ECS cluster + Fargate services (`backend`, `worker`, `ui`, `codesearch`) | Runs application APIs, background jobs, UI, and code indexing/search workloads. | | Chat sandboxes | One EC2 Docker host in an Auto Scaling group, Docker volume, TLS secrets, CloudWatch alarms | Runs each Workspace conversation's agent in its own container. | | Stateful data | Aurora PostgreSQL 16, Neptune cluster, EFS for `/data` | Persists relational state, graph data, and codesearch cache/indexes. | | Knowledge store | One EC2 ClickHouse host in an Auto Scaling group, an EBS data volume, an S3 bucket for old parts | Holds the content of the [new knowledge store](/docs/self-hosting/configuration#knowledge-store). | | Secrets + identity | Secrets Manager secrets, generated `AUTH_SECRET`, optional connector secrets | Stores runtime credentials and injects them into tasks securely. | | Email delivery | SES domain identity, DKIM, SMTP credentials | Enables transactional email from backend (`ctxpipe-noreply@`). | | Deploy safety | Migration custom resource (one-off ECS task) + ECS deployment rollback controls | Runs migrations during deploy and prevents partial/broken rollouts. | Only the backend is public. UI and codesearch stay internal and are reached through backend routing. ## Knowledge store `CtxPipe` deploys ClickHouse 26.10 for the [new knowledge store](/docs/self-hosting/configuration#knowledge-store). The new knowledge store is off for every organization. While it is off, Neptune answers, and the backend and the worker do not use ClickHouse. ### What it sets up | Area | What the construct does | | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Host | One Graviton EC2 instance in an Auto Scaling group of one, in the first private subnet. ClickHouse runs in Docker on the host network. The image is the same pinned image as in Docker Compose. | | Data | An encrypted gp3 EBS volume holds the new parts and the cache of the old parts. The volume is not part of the instance: each new host attaches it again. When you delete the stack, CloudFormation keeps a snapshot of the volume. | | Old parts | The knowledge tables move a part to the volume `cold` when its newest row is 30 days old. On AWS, `cold` is an S3 bucket of the stack behind a cache on the EBS volume (a fifth of the volume). The bucket blocks public access, is encrypted, and stays when you delete the stack. When the volume has less than 20% free space, ClickHouse also moves its largest parts to the bucket. The VPC has an S3 gateway endpoint, so this traffic does not go through the NAT gateway. | | Network | Only the backend, the worker, and the migrate task reach ClickHouse, on port 8123 (HTTP), at `clickhouse.ctxpipe.local`. The host registers this Cloud Map name when it boots. There is no public listener and no SSH key. | | Users | The stack generates three passwords in Secrets Manager. The migrate task gets the admin user and the two other passwords, and makes the users `ctxpipe_reader` and `ctxpipe_writer`. The backend gets `ctxpipe_reader` (read only). The worker gets `ctxpipe_writer`. Each task gets its own `CLICKHOUSE_URL`. | | Credentials | ClickHouse reads and writes the bucket with the instance role. The role reaches only the prefix `clickhouse/` of the bucket. | | Logs | ClickHouse logs to CloudWatch Logs (14 days). Of the server's own log tables, only `system.query_log` and `system.error_log` stay, for 7 days. | The output `KnowledgeStoreAutoScalingGroupName` names the group. Open a shell on its instance with Session Manager for checks. ### Knowledge store alarm | Alarm | Fires when | What to do | | ------------ | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Status check | The ClickHouse host fails its EC2 status checks for 2 minutes | The Auto Scaling group replaces the host, and the new host attaches the same data volume. If the alarm stays on, look at the group's activity history and the ClickHouse log group. | The alarm has no actions. Subscribe it to your own topic through `ctxPipe.knowledgeStoreAlarms`, as for the [chat sandbox host alarms](#alarms). The host has EC2 detailed monitoring, because EC2 publishes the status checks by Auto Scaling group only with it. ### Sizes and cost | `size` | Instance | vCPU / RAM | Data volume (gp3) | \~Cost/month | | ----------------- | ------------ | ---------- | ----------------- | ------------ | | `small` (default) | `t4g.large` | 2 / 8 GiB | 50 GB | \~$55 | | `medium` | `r7g.large` | 2 / 16 GiB | 100 GB | \~$88 | | `large` | `r7g.xlarge` | 4 / 32 GiB | 200 GB | \~$174 | Costs are us-east-1 on-demand prices for the instance, the data volume, and a 16 GB root volume. The S3 storage of the old parts and the detailed monitoring of the host (a few dollars a month, for the status check alarm) add to this. A move to a larger `size` replaces the host and makes the volume larger. Do not move to a smaller `size`: EBS cannot make a volume smaller, and the deploy fails. ### Recovery ClickHouse holds only content that ctxpipe makes from the Workspace repositories. A replacement host attaches the same volume, so it keeps the data. If the volume is lost, follow [Recover the knowledge store](/docs/self-hosting/configuration#recover-the-knowledge-store). On AWS, the next `cdk deploy` that changes the migrate task runs the migrations. A single host is a single point of failure: while it is replaced, the new knowledge store does not answer. ## Chat sandbox host Workspace chat runs each conversation's agent (OpenCode) in its own Docker container. `CtxPipe` creates one EC2 host for those containers by default. The sandbox keeps the agent's commands away from the backend and your data stores. To create no host, set `sandboxHost: { enabled: false }`. Without the host, Workspace chat is off. The product answers chat requests with a "sandbox unavailable" error (503). The rest of the product keeps working. ### Sizes and cost The host mostly holds idle sandboxes, so it uses burstable Graviton (ARM) instances. Memory is the limit. An active sandbox uses about 0.3–0.5 GiB. Sandboxes stop after 5 minutes idle and resume on the next message. Stopped sandboxes are removed 30 days after last use. An organization runs at most 50 sandboxes at once. | `size` | Instance | vCPU / RAM | Docker volume (gp3) | \~Cost/month | Running sandboxes (est.) | | ----------------- | ------------ | ---------- | ------------------- | ------------ | ------------------------ | | `small` (default) | `t4g.medium` | 2 / 4 GiB | 30 GB | \~$28 | \~6 | | `medium` | `t4g.large` | 2 / 8 GiB | 50 GB | \~$54 | \~14 | | `large` | `t4g.xlarge` | 4 / 16 GiB | 100 GB | \~$107 | \~30 | Costs are us-east-1 on-demand prices for the instance, the Docker volume, and a 16 GB root volume. `t4g` instances run in unlimited mode, so sustained CPU above baseline adds a small surcharge. Override either value with `sandboxHost`: ```ts import * as ec2 from "aws-cdk-lib/aws-ec2"; new CtxPipe(stack, "CtxPipe", { // ...orgSlug, customDomain, modelProvider sandboxHost: { instanceType: ec2.InstanceType.of(ec2.InstanceClass.T4G, ec2.InstanceSize.XLARGE), dockerVolumeSizeGiB: 150, }, }); ``` The instance type must be Graviton (arm64). Synth rejects other architectures. ### Private git servers Ingestion can read a plain git repository on a git server inside your network (see [Plain git repositories](/docs/workspaces/linked-repositories#plain-git-repositories)). The ingestion sandbox fetches it through Agent Vault on the sandbox host. Agent Vault connects only to the private addresses in its allowlist. By default, the allowlist holds only the backend subnets. Add the single host address of your git server with `sandboxHost.agentVaultNetworkAllowlist`: ```ts new CtxPipe(stack, "CtxPipe", { // ...orgSlug, customDomain, modelProvider sandboxHost: { agentVaultNetworkAllowlist: ["10.20.0.15/32"], }, }); ``` Chat-sandbox vaults forward each host that no rule matches. An address or CIDR in this list therefore lets every chat agent reach every port and path at that address. Add only the single host address of the git server. Never add a whole subnet. The stack adds each entry after the backend subnets. Each entry must be an IPv4 or IPv6 address or CIDR. Synth rejects an entry with a cloud metadata or link-local address. The sandbox host must also have a route to that address, for example VPC peering or a transit gateway. The git server must accept connections from the host. A change to the list replaces the host, as each change to its boot script does. ### What it sets up | Piece | Behavior | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Instance | Amazon Linux 2023 in a single-instance Auto Scaling group in the private subnets. EC2 health checks replace a failed host. `cdk deploy` waits until the host reports ready. New AMIs apply when the host is replaced, not on every deploy. | | Docker volume | gp3 volume at `/var/lib/docker`. It belongs to the instance, so a replaced host starts empty. Conversations keep their work in their git branch and continue on the new host. | | Docker API | Port 2376 with mutual TLS at `sandbox-host.ctxpipe.local`, a Cloud Map name the host registers when it boots. The host creates the certificates on first boot and stores them in Secrets Manager; a replacement host reuses them. The CA key is not kept. | | Network | Backend and worker reach the Docker API and the Agent Vault API (port 14321). Only the backend reaches the sandboxes' agent ports. `DOCKER-USER` rules let sandboxes reach only the Agent Vault proxy (port 14322); a direct connection, a non-HTTP protocol, and DNS are rejected. Docker forwards the proxy port without NAT, so Agent Vault sees the address of each sandbox: failed proxy logins from one sandbox do not block the proxy for the other sandboxes. Through the proxy, sandboxes reach Git hosts, the internet, and the backend (model proxy and per-run tool bridges), but not Postgres, Neptune, ClickHouse, EFS, codesearch, the UI, or the worker. This is the [chat sandbox network policy](/docs/self-hosting/architecture#chat-sandbox-network-policy) shared with Docker Compose. | | Credentials | Agent Vault (`infisical/agent-vault:latest`) runs on the host on its own Docker bridge. Sandboxes hold no credential: Agent Vault adds the GitHub token and the backend credentials to each request. Agent Vault may dial the backend's subnets, and the entries of `sandboxHost.agentVaultNetworkAllowlist`. The backend subnets are the stack's only private subnets, so they also hold RDS, Neptune, the ClickHouse host, EFS, and the UI and codesearch tasks; only security groups keep the proxy out of them. If you change security groups, never admit the sandbox host's group to those services. The stack generates its master password and owner password in Secrets Manager; the host reads the master password, and backend and worker get the owner password. If Agent Vault does not answer, chat fails closed. A replaced host starts an empty Agent Vault, and the backend registers again on first use. | | Isolation | Each agent port needs a per-conversation password. Agent ports are reachable inside the VPC, so the host also blocks sandboxes from each other, from the host itself, and from instance metadata. | | Limits | All sandbox containers share one systemd slice capped at 85% of host memory. Container logs rotate at 3 × 10 MB. | | Chat image | `ghcr.io/ctxpipe-ai/chat-sandbox`, published for arm64 and amd64 with the same release tag as the service images. The backend pulls it through the host's Docker daemon on the first chat after a release, so releases never replace the host. | | Backend and worker | Get `SANDBOX_PROVIDER=docker`, `SANDBOX_CHAT_IMAGE`, `DOCKER_HOST=tcp://sandbox-host.ctxpipe.local:2376`, `DOCKER_TLS_VERIFY=1`, and `DOCKER_CERT_PATH`. A short init container in each task writes the client certificates there as files. They also get `AGENT_VAULT_ADDR` and the `AGENT_VAULT_OWNER_PASSWORD` secret. Sandboxes call the backend back at its task's VPC DNS name (`ip-….compute.internal`), so they reach the replica that runs their turn. The stack sets `SANDBOX_CALLBACK_DNS_SUFFIX` for this. The VPC must have DNS host names and DNS resolution on; the stack's VPC has both. | ### Alarms | Alarm | Fires when | What to do | | ----------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Docker disk | `/var/lib/docker` is 80% full or more for 10 minutes | Check `docker system df` on the host (see [Operations](/docs/self-hosting/operations#chat-sandbox-host-aws)). Raise `sandboxHost.dockerVolumeSizeGiB` if the volume fills with live sandboxes. | | Memory | Host memory is 85% used or more for 15 minutes | Sandboxes are near their cap. Move to a larger `size` or `sandboxHost.instanceType`. | The alarms have no actions. Subscribe them to your own topic through the construct: ```ts import * as actions from "aws-cdk-lib/aws-cloudwatch-actions"; const ctxPipe = new CtxPipe(stack, "CtxPipe", { /* ... */ }); for (const alarm of ctxPipe.sandboxHostAlarms) { alarm.addAlarmAction(new actions.SnsAction(opsTopic)); } ``` The backend and worker images refuse `SANDBOX_PROVIDER=unsandboxed`. An agent without a sandbox would run with the backend's credentials. `CtxPipe` configures a Docker host for chat sandboxes. ## What this gives you by default * `GRAPH_DB_PROVIDER=neptune` and graph URI wiring from Neptune endpoints * Internal service routing (`UI_PROXY_URL`, `CODESEARCH_URL`) so only backend is public * Generated `AUTH_SECRET` and managed `DATABASE_URL` (runtime `ctxpipe_app`) / SMTP secrets * A `CLICKHOUSE_URL` for each of backend, worker, and migrate, each with its own ClickHouse user * Public app URL output and secret ARN outputs for operations ## How to choose `size` Use `size` to tune compute and database capacity for your single-tenant workload: * `small` (default): for pilot and small-team deployments with moderate repository churn and cost sensitivity. Aurora and Neptune both use `db.t4g.medium`. This is the minimum burstable size that AWS supports for these engines (for dev and test). Codesearch is 4 GiB (ingest peak, not idle RSS). * `medium`: for sustained ingestion and reindex workloads where the worker and codesearch need more headroom. Codesearch is 8 GiB. Aurora uses `db.t4g.large`; Neptune uses memory-optimized `db.r6g.large`. * `large`: for high ingestion and lower latency. It has bigger tasks, scaled backend and worker replicas, and `db.r6g.xlarge` for both Aurora and Neptune. Codesearch is 12 GiB and still **one task**. The two workers each get `OPENWORKFLOW_CONCURRENCY=8`, so the cluster cannot overload codesearch. Extra workers help extract and connectors. They do not help clone or SCIP parallelism. See the [package README sizing table](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#sizing-profiles) for the full preset matrix. Do **not** raise the desired count of codesearch. Ingest throughput grows with codesearch memory (4 / 8 / 12 GiB) and the environment variables that the construct injects (`OPENWORKFLOW_CONCURRENCY`, `CODESEARCH_INDEXER_CONCURRENCY`, `CODESEARCH_INDEX_PIPELINE_CONCURRENCY`). Signals that you should move up a size: * ingestion jobs queueing up or taking too long, * frequent codesearch reindex delays, * higher p95 response latency during active sync/index windows. `@ctxpipe/aws-cdk` keeps services in private subnets for every size. You still need NAT, because the backend, worker, and codesearch need outbound access to model providers and to the GitHub, Atlassian, and Linear APIs. ## Related references * [`packages/aws-cdk/README.md`](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md) * [`examples/aws-cdk-self-host`](https://github.com/ctxpipe-ai/ctxpipe/tree/main/examples/aws-cdk-self-host) # Docker (https://docs.ctxpipe.ai/docs/self-hosting/deployment/docker) Docker Compose is the reference self-hosted path. Use it for an evaluation, for production-style tests, and for a small internal deployment on one host. ## Start the stack You need Docker with Compose, Node.js, and pnpm. Run these commands from a clone of the repository: ```bash git clone https://github.com/ctxpipe-ai/ctxpipe.git cd ctxpipe cp docker-compose.env.example .env ``` Set at least these values in `.env`: ```bash AUTH_SECRET=<32-plus-character-secret> AUTH_BASE_URL=https://ctx.example.com CTXPIPE_PUBLIC_APP_URL=https://ctx.example.com MODEL_PROVIDER=openrouter MODEL_PROVIDER_API_KEY= ``` Then start the deploy profile: ```bash pnpm start ``` `pnpm start` runs `docker compose --profile deploy up -d`. It builds the images only when they do not exist. For a local evaluation, `AUTH_BASE_URL` and `CTXPIPE_PUBLIC_APP_URL` can be `http://localhost:3000`. For team use, put a TLS proxy in front of the backend and use the public HTTPS origin. The UI image contains `CTXPIPE_PUBLIC_APP_URL` at build time. After you change it, build the images again with `docker compose --profile deploy up -d --build`. ## What the deploy profile starts | Service | Purpose | | --------------------- | -------------------------------------------------------------------------------------- | | `migrate` | Runs the database migrations as the Postgres owner, then gives access to `ctxpipe_app` | | `backend` | The public API, auth, MCP, callbacks, and the UI proxy. Published on host port `3000`. | | `ui` | The web app, served through the backend | | `worker` | The background jobs for ingestion and connectors | | `codesearch` | The repository clone cache and the Zoekt indexes. Keep one replica. | | `postgres` | The application database | | `falkordb` | The default graph database | | `clickhouse` | The knowledge content | | `dind` | The Docker daemon for chat and ingestion sandboxes | | `chat-sandbox-image` | One-shot: builds the sandbox image inside `dind` | | `agent-vault` | Adds credentials to sandbox requests. It is the only address that sandboxes can reach. | | `agent-vault-secrets` | One-shot: generates the Agent Vault passwords | The backend is the only service on a public host port (`3000`). Compose binds `codesearch` (`3001`) and `ui` (`3002`) to `127.0.0.1` only. Postgres, FalkorDB, and ClickHouse have no host port. Compose gives each service its own ClickHouse user in `CLICKHOUSE_URL`: | Service | ClickHouse user | Password in `.env` | Access | | --------- | ----------------- | ---------------------------- | ------------------------------------------------------------------------------------------------- | | `migrate` | `CLICKHOUSE_USER` | `CLICKHOUSE_PASSWORD` | Admin. It makes the tables and the two other users. | | `backend` | `ctxpipe_reader` | `CLICKHOUSE_READER_PASSWORD` | Reads the knowledge tables only | | `worker` | `ctxpipe_writer` | `CLICKHOUSE_WRITER_PASSWORD` | Reads, inserts, and deletes rows in the knowledge tables. It cannot manage users or row policies. | Set `CLICKHOUSE_READER_PASSWORD` and `CLICKHOUSE_WRITER_PASSWORD` before the first start. They have no default. Without them, `migrate` does not make the two users, and the backend and the worker cannot connect to ClickHouse. The default admin user and password are `ctxpipe`. Set your own `CLICKHOUSE_PASSWORD` too. `migrate` refuses the reader and writer passwords when the admin password is `ctxpipe`, or when two of the three passwords are the same. Use only URL-safe characters in each password. `CLICKHOUSE_DB` sets the database (default `ctxpipe`). To change a reader or writer password, change it in `.env` and run `migrate` again. ## Settings that need an override file The deploy profile passes only a fixed list of variables from `.env` to each service. The list has the auth, model, OpenTelemetry, and sandbox settings, and the Slack, Linear, Notion, and PagerDuty connector secrets. Other values in `.env` do not reach the services. Add them to the `environment` of `backend` and `worker` in a `docker-compose.override.yml`. This applies to: * email: `SMTP_CONNECTION_URL` and `EMAIL_FROM_ADDRESS`. Without them, ctxpipe sends no verification, password reset, or invitation email; * social sign-in: `GITHUB_CLIENT_ID`, `GOOGLE_*`, and `MICROSOFT_*`; * Atlassian OAuth: `ATLASSIAN_CLIENT_ID` and `ATLASSIAN_CLIENT_SECRET`; * an external graph database: `GRAPH_DB_URI`, `GRAPH_DB_PROVIDER`, `GRAPH_DB_USER`, and `GRAPH_DB_PASSWORD`. ```yaml services: backend: environment: SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL} EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS} worker: environment: SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL} EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS} ``` An S3 disk for the old ClickHouse parts also needs an override file: mount your storage file on the `clickhouse` service. See [Keep old parts on S3](/docs/self-hosting/configuration#keep-old-parts-on-s3). The in-product GitHub wizard stores the GitHub App credentials in the database. The GitHub App therefore needs no variable. ## Persistent state The deploy profile mounts no named volume for `postgres` or `falkordb`: * Postgres keeps its data in an anonymous Docker volume. `docker compose down -v` deletes it. After `docker compose down`, the new container does not use the old volume. * The FalkorDB image declares no volume, so FalkorDB keeps its data in the writable layer of the container. Each time Compose creates the container again, you lose the graph data. This happens, for example, after `down`, an image update, or a configuration change. Before people depend on the deployment, mount named volumes in `docker-compose.override.yml`: ```yaml services: postgres: volumes: - postgres_data:/var/lib/postgresql/data falkordb: volumes: - falkordb_data:/var/lib/falkordb/data volumes: postgres_data: falkordb_data: ``` Do this before the first start. On a running deployment, first back up the data. Then restore it into the new volumes. | State | Where it is | Back it up | | ------------------------------ | --------------------------------------------------------------------- | -------------------------------------------------------------- | | Postgres data | An anonymous volume of the `postgres` container, or your named volume | Yes | | Graph data | The `falkordb` container, or your named volume | Yes, or rebuild it from the Workspace repositories | | Knowledge content | `clickhouse_data` | Yes, or rebuild it from the Workspace repositories | | Repository clone cache | `repo_cache` | No, codesearch clones again | | Zoekt indexes | `zoekt_index_deploy` | No, codesearch indexes again | | Agent Vault passwords and data | `agent_vault_master`, `agent_vault_owner`, `agent_vault_data` | Keep them. New passwords do not match an existing Agent Vault. | | Sandboxes and TLS certificates | `sandbox_docker`, `sandbox_ca`, `sandbox_client_certs` | No | | Secrets | `.env`, or your secret store | Yes | To back up Postgres, run `docker compose exec postgres pg_dump -U ctxpipe ctxpipe > backup.sql`. ### Database passwords Compose sets two Postgres URLs for you. `migrate` connects as `POSTGRES_USER` (default `ctxpipe`). The other services connect as `ctxpipe_app`. Both use `POSTGRES_PASSWORD`. To change the password: 1. Change the owner password in Postgres: `docker compose exec postgres psql -U ctxpipe -c "ALTER ROLE ctxpipe PASSWORD ''"`. Postgres reads `POSTGRES_PASSWORD` only when it creates the volume. 2. Set the same value as `POSTGRES_PASSWORD` in `.env`. 3. Run `docker compose --profile deploy up -d`. `migrate` sets the new `ctxpipe_app` password, and the services restart with it. ## Chat sandboxes Each conversation and each ingestion runs in its own container in `dind`, a Docker-in-Docker daemon. It needs no settings. * `dind` is the only privileged container. The sandboxes are not privileged, and the backend never mounts the Docker socket of the host. * The Docker API listens on port `2376` inside Compose, with mutual TLS. It is not published on the host. * Sandboxes hold no credential. They can reach only Agent Vault. See the [sandbox network policy](/docs/self-hosting/architecture#chat-sandbox-network-policy). * `dind`, `agent-vault`, `backend`, and `worker` share the `sandbox` network, `172.30.99.0/24`. If it overlaps a network of your host, change each `172.30.99.x` address in `docker-compose.yml`. * `chat-sandbox-image` builds the sandbox image on each `pnpm start`. Until the image exists, chat answers `503`, and the rest of the app works. * On a cgroup v2 host, all sandboxes share one limit of 8192 processes. Without cgroup v2, `dind` logs a warning and applies no limit. * Container logs rotate at 3 files of 10 MB. * If you remove `sandbox_ca`, `dind` makes new certificates. Then restart `backend` and `worker`. To limit the memory of all sandboxes, limit `dind` in `docker-compose.override.yml`. The sandboxes together get at most 85% of it: ```yaml services: dind: mem_limit: 8g ``` Each process that has the files in `sandbox_client_certs` can start privileged containers on `dind`. That access is root on the Docker host. Do not mount that volume in another service. Do not copy it off the host. If `dind` stops, chat answers `503` until it is back. It never runs an agent without a sandbox. The backend and worker images refuse `SANDBOX_PROVIDER=unsandboxed`, because an agent without a sandbox has the credentials of the backend. If your host cannot run privileged containers, chat is not available. To read a plain git repository on a private git server, see [Plain git repositories](/docs/self-hosting/configuration#plain-git-repositories). ## Check the deployment * `docker compose --profile deploy ps` shows healthy services and a healthy `dind`. * `curl https:///.status` returns `ok`. * You can sign in through the public origin. * A linked repository reaches **Indexed**. * Workspace chat answers, and `docker compose exec dind docker ps` shows its sandbox. * An MCP client reaches `{AUTH_BASE_URL}/mcp`. See [Operations](/docs/self-hosting/operations#chat-sandboxes-docker-compose) for the sandbox disk and health checks. # Buildkite connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/buildkite) The **Buildkite connector** writes the latest builds of selected pipelines into a [Workspace's](/docs/workspaces) GitHub repository. ctxpipe then indexes them like other knowledge files. An agent can see which builds failed, which jobs failed, and the commit and pull request of each build. A source file in the Workspace holds the pipeline scope. A pull request adds it, so a person reviews the scope before content syncs. ctxpipe only reads from Buildkite. It does not start, retry, or cancel builds. ## What ctxpipe syncs You select pipelines for each Workspace. For each pipeline, ctxpipe writes: * `pipeline.md`: the name, repository, default branch, and link. * One file for each of the latest 25 finished builds: the state, branch, commit, pull request, the person who created the build, and the jobs that failed, with a link to each. A soft-failed job does not fail the build, so it is not listed. A finished build is passed, failed, canceled, or blocked. A blocked build waits at a block step. When someone unblocks it, the build continues and is no longer finished. The next sync deletes its file. The file comes back when the build finishes. ctxpipe does not read build logs, environment variables, meta-data, or the commands of steps. These can contain secrets. When the repository of a pipeline is on GitHub, a build file links to its commit and pull request. The Workspace graph then joins the build to them. ## Prerequisites * A Buildkite user that can read the pipelines of the organization. * A [Workspace](/docs/workspaces) whose repository the ctxpipe GitHub App can write to. The repository must use format 2 or higher. If it does not, the dialog shows **Needs the format upgrade first.** next to the Workspace. See [Upgrades](/docs/workspaces/settings#upgrades). * Permission to merge the pull request that adds the source. * Optional: permission to add a webhook in the Buildkite organization settings. ## Where the scope lives The connection holds the tokens only. The scope is in a source file in each Workspace, at `sources/buildkite/.md`. The front matter holds the pipeline slugs. The body is an optional description of what the source is for. ```markdown --- provider: buildkite connection: con_example pipelines: [web-app, deploy-production] --- ``` The scope goes live after the merge. See [Workspace settings](/docs/workspaces/settings#sources) for draft scope and live scope. Each Workspace has its own scope. One source holds at most 50 pipelines. ## Setup flow ### Create an API access token In Buildkite, open [API Access Tokens](https://buildkite.com/user/api-access-tokens/new). Select only your organization. Select only these REST scopes: `read_pipelines` and `read_builds`. Do not select **Enable GraphQL API access**. That option gives the token full access, including write access. ctxpipe refuses a token that has any other scope or GraphQL API access, because the connector only needs these two scopes. ### Connect Buildkite Open **Connectors**, select **Add connection**, then select **Connect** on the **Buildkite** row. Type the organization slug (the name in your Buildkite URL) and paste the token. ctxpipe checks the token with Buildkite before it saves it. ctxpipe keeps the token encrypted and never shows it again. ### Add the webhook The webhook tells ctxpipe when a build finishes. In Buildkite, open **Settings → Notification Services**, and add a **Webhook**: 1. **Webhook URL:** paste the URL that the setup shows. 2. **Token:** select the signature option (`X-Buildkite-Signature`). ctxpipe refuses a plain token. 3. **Events:** clear every event except `build.finished`. 4. **Pipelines:** keep all pipelines, or select the pipelines that your Workspaces sync. Then save the webhook. 5. Copy the token of the webhook, and paste it into the setup. To add the webhook later, close the setup. The connector card shows **Add the webhook**, and **Add to a Workspace** under **Used in**. Until then, ctxpipe syncs once a day and when you select **Sync now**. ### Add to Workspaces The next step is **Add Buildkite to Workspaces**. 1. Select the Workspaces that sync from Buildkite. 2. Select at least one pipeline. Type in the filter to find a pipeline by its name or slug. 3. If you select more than one Workspace, choose **Same scope for each Workspace** or **A different scope for each Workspace**. 4. Choose **Add to Workspaces**. If the organization has no Workspace yet, the step shows **Create a Workspace**. Create the Workspace, then return to **Connectors** and choose **Add to a Workspace** on the Buildkite card. The connection stays saved. You can add the connection to a Workspace later. Expand the connector on **Connectors**. When no Workspace uses the connection, choose **Add to a Workspace**. Otherwise, choose **Manage**. ### Merge the pull requests ctxpipe opens one pull request for each Workspace. Each pull request adds `sources/buildkite/.md`. The dialog shows **Merge the pull request** (or **Merge the pull requests**) with a link to each one. 1. Review the selected pipelines and the repository audience. 2. Merge each pull request. After each merge, the dialog shows **Pull request merged**. The source does not sync before the merge. ### Wait for the first sync After GitHub reports the merge, ctxpipe reads the source file and writes the files of each pipeline. The Workspace indexes the new files after that. See [Indexing](/docs/workspaces/indexing) for the lifecycle. ## Change what syncs Edit, rename, or remove a source in the **Settings** of the Workspace. See [Workspace settings](/docs/workspaces/settings#sources). When you remove a pipeline from the scope, ctxpipe deletes the files of that pipeline in a sync commit. ## Files in the repository ```text buildkite/pipelines//pipeline.md buildkite/pipelines//builds/.md ``` Do not edit these files. ctxpipe writes them again on each sync. When a build is no longer among the latest 25, ctxpipe deletes its file in the same commit. ## How builds stay current * **Webhook:** a signed `build.finished` event starts a sync of the connection at the end of a 10-minute window. More events in the same window start no more syncs, so a busy pipeline gives at most one commit for each Workspace in each window. * **Daily sync:** ctxpipe syncs each connection once a day. * **Sync now:** an organization owner or admin can select **Sync now** in the menu of the connector card. ## Verify the complete flow 1. Connect Buildkite and confirm that the organization slug appears on **Connectors**. 2. Add the webhook, and confirm that the card does not show **Add the webhook**. 3. Choose **Add to Workspaces**, select a Workspace, and select at least one pipeline. 4. Merge the pull request that adds `sources/buildkite/.md`. 5. Confirm that the Workspace repository has files under `buildkite/pipelines/`. 6. Finish a build in a selected pipeline. Confirm that its build file appears at the end of the 10-minute window. 7. Confirm that the Workspace index is ready in its **Settings**. Steps 4 and 5 test the GitHub push webhook and the first sync. Step 6 tests the Buildkite webhook delivery. ## Troubleshooting ### Buildkite refuses the token The setup shows the reason. Make sure that the token has the two read scopes and no GraphQL API access. Make sure that the organization of the token is the organization slug that you typed. If the setup says that ctxpipe cannot make sure that the token has no GraphQL API access, ask the operator to check that the backend can reach `graphql.buildkite.com`. ### Add to a Workspace shows an error The dialog shows a message about the format, the write access, or your role. See [Troubleshooting](/docs/connections#troubleshooting). ### No pull request appears Confirm that the GitHub App has write access to the Workspace repository. If the dialog reports an error, close it and try again. ### The source shows a status other than OK Read the status and the reason of the row in **Settings**. See [Source status](/docs/workspaces/settings#source-status). ### Content sync failed Choose **Sync now** in the menu of the connector card. If it fails again, ask the operator to check the worker logs, GitHub write access, and outbound access to `api.buildkite.com`. ### The card shows “Needs reconnect” Buildkite refused the saved token, for example because someone deleted it. The card says "Buildkite does not accept the saved token. Reconnect with a new token." The source shows the reason "Buildkite does not accept the API access token of this connection any more. Reconnect Buildkite on the Connectors page." ctxpipe keeps the files and stops the sync. Select **Reconnect Buildkite** on the card and paste a new token. ### A source shows “Out of access” A selected pipeline was deleted or renamed, or the token cannot read it. ctxpipe tries the pipeline in three sync cycles and keeps its files. Then it marks the source. Remove the pipeline from the source, or select it again by its new name. ### Builds do not arrive after they finish Make sure that the webhook uses the signature option, sends `build.finished`, and includes the pipeline. A build arrives at the end of the 10-minute window of its event. The webhook settings in Buildkite show the response of each delivery. A `401` response means that the token in ctxpipe is not the token of the webhook. Expand the connector card, select **Replace the webhook token**, and paste the token again. ### Remove the connector Choose **Remove connector** in the menu of the connector card. See [How content stays current](/docs/connections#how-content-stays-current) for what stays in Git. This deletes the saved tokens. The Workspace sources that use the connector stop syncing. Delete the webhook in Buildkite too, and the API access token if nothing else uses it. # Confluence connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/confluence) The **Confluence connector** mirrors the Confluence spaces and pages that you select into a [Workspace](/docs/workspaces) repository. ctxpipe does not read the whole site. You choose what agents can use. ctxpipe connects to the Confluence site through Atlassian OAuth and the ctxpipe Forge app. When pages or spaces change, the Forge app tells ctxpipe, and ctxpipe updates the mirror. On a self-hosted deployment, the operator provides the Atlassian OAuth app and the Forge setup first. See [Self-host Confluence & Atlassian](/docs/self-hosting/confluence-atlassian). ## Data flow ## Before you start You need: * permission to authorize Atlassian OAuth for the Confluence site; * permission to install the ctxpipe Forge app on the site, or an Atlassian admin who can install it. Open `admin.atlassian.com`, choose the Confluence site, and check whether you can open **Connected apps** and install apps. Atlassian requires organization admin or site admin access for this. If you cannot, ask an Atlassian admin to install the Forge app. See [Atlassian Connected apps](https://support.atlassian.com/organization-administration/docs/managing-an-installed-app/). ## Connect Confluence ### Start from Connectors Open **Connectors**, choose **Add connection**, then **Connect** on the **Confluence** row. The setup wizard tracks each task until the connector is ready. ### Link your Atlassian account Link your Atlassian account with OAuth. This tells ctxpipe which Atlassian cloud site you connect. If the account is already linked to another ctxpipe organization, the wizard asks you to confirm or cancel. ### Install the Forge app Install the ctxpipe Forge app on the same Atlassian site. The Forge app lets ctxpipe read the content that you select, and tells ctxpipe about changes. ### Add to Workspaces Select the Workspaces, then select the spaces and pages in the tree. For each space, select all pages or only some. Choose the scope with care: * start with the spaces that your engineering team uses; * leave out old or sensitive areas that agents must not rely on; * choose the authoritative docs, not duplicates or abandoned pages. Choose **Add to Workspaces**, then merge the pull request that adds `sources/confluence/.md`. See [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace). ## Source file For each space, `pages` is `all` or a list of page ids: ```markdown --- provider: confluence connection: con_example spaces: - key: ARCH pages: all - key: OPS pages: ["12345", "67890"] --- ``` ## Files in the repository Mirrored pages are under `confluence//`, in the page tree of the space. A page is `--.md`, or `--/index.md` when it has child pages. Attachments and embedded media are under `confluence//_assets//`. Each asset can be up to 25 MiB, and each page up to 100 MiB in total. A file that is too large or not safe stays a link to Confluence. ctxpipe never writes Atlassian download URLs to git. For problems, see [Troubleshooting](/docs/connections#troubleshooting). # GitHub connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/github) GitHub is the first connector to set up. The ctxpipe GitHub App gives ctxpipe access to your [Workspace](/docs/workspaces) repositories and to the code repositories that you link. The other connectors, including Buildkite, write their content into a Workspace repository on GitHub, so they need it too. Hosted ctxpipe supplies the GitHub App. A self-hosted deployment creates and owns its own App. The operator completes [Self-host GitHub](/docs/self-hosting/github) first. ## Connect GitHub ### Start the connection Choose **Connect GitHub** in the **Get started** guide on Home, or open **Connectors** and choose GitHub. ### Install the App Install the ctxpipe GitHub App on the GitHub account or organization that owns your repositories. If you cannot install apps there, GitHub sends a request to an organization owner. After the install, GitHub can ask you to authorize the ctxpipe App. This shows ctxpipe that your GitHub account can access the installation. ctxpipe does not keep this authorization, and it does not link your GitHub account to your ctxpipe user. ### Choose the repositories Give the App access to the repository for the knowledge and to the code repositories that you want to link. You can start with a few repositories, and add more later in GitHub. ### Create a Workspace and link code [Create a Workspace](/docs/workspaces/create-workspace), then [link code repositories](/docs/workspaces/linked-repositories) in its **Settings**. ## What GitHub enables * Workspace repositories that ctxpipe can read and write. * Code search and extraction across the linked repositories. * The docs, ADRs, `AGENTS.md`, and skills in linked repositories, as context for agents. * Pull requests for source changes, upgrades, and Workspace chat. * A mirror of the issues and the closed pull requests of each linked repository. ## Closed pull requests When you link a GitHub repository, ctxpipe also mirrors its closed pull requests, merged or not, into `github/pulls///` in the Workspace repository. Each file holds the conversation, the reviews, and the changed paths, but not the diff. It records `merged` and `mergedAt`. * **When you link** the repository, ctxpipe backfills up to 200 of its most recently updated closed pull requests. * **After that**, each pull request that closes, and each later review or comment, arrives when GitHub reports it. * **Each Workspace** that links the repository gets its own copy. * **When you unlink**, new pull requests stop. The mirrored files stay until you delete them. Draft and open pull requests are not mirrored. There is no setting for this. ## Issues ctxpipe also mirrors the issues of each linked repository, open and closed. Each issue is one file at `github/issues///.md`, with its description, labels, assignees, state, and its 100 most recent comments. * **When you link** the repository, ctxpipe backfills up to 200 of its most recently updated issues. * **After that**, each change to an issue, and each comment on it, arrives when GitHub reports it. When someone deletes an issue or moves it to another repository, ctxpipe removes its file. The GitHub App needs the **Issues: Read-only** permission. Until the owner of your GitHub installation accepts that permission, ctxpipe does not mirror issues. Pull requests and code continue to work. When the owner accepts it, ctxpipe backfills the issues of each linked repository of that account. ## In the graph A pull request links to its repository, to the GitHub issues whose URL it contains, and to the Linear issues that it names. A merged pull request also links to each file that it added, changed, removed, or renamed. A pull request **resolves** an issue when its branch names the issue. It also resolves an issue when a closing word, such as `fixes` or `closes`, comes before the issue identifier. A GitHub issue links to the pull requests that closed it. # Linear connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/linear) The **Linear connector** mirrors selected Linear teams and projects into a [Workspace](/docs/workspaces) repository. ctxpipe requests read-only access. It does not create or change Linear issues. On a self-hosted deployment, the operator registers a Linear OAuth app first. See [Self-host Linear](/docs/self-hosting/linear). ## What is mirrored A team syncs its issues, projects, cycles, and labels. A project syncs its issues from all teams. The mirror can include: * issues and their comments, with the parent issue, the project milestone, and the relations to other issues (blocks, duplicate, related); * project and initiative updates; * cycles, labels, and the users that the content mentions; * customer requests, unless you clear **Include customer requests**; * attachments and embedded media, copied into git. ctxpipe does not copy GitHub pull requests, commits, or diffs through Linear. It keeps only the link. The GitHub connector already indexes that content. Customer requests can contain the names and feedback of your customers. Each person who can read the Workspace repository can read them. Review the scope before you merge the pull request. ## Connect Linear Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace). For Linear: 1. On **Connectors**, select **Add connection**, then select **Connect** on the **Linear** row. Authorize the ctxpipe app in the Linear popup. 2. In **Add to Workspaces**, select the Workspaces, then the teams and projects. Clear **Include customer requests** to leave them out. 3. Merge the pull request that adds `sources/linear/.md`. To change the Workspaces or the scope later, choose **Manage** on the connector card. ## Source file ```markdown --- provider: linear connection: con_example teams: [ENG, OPS] customerRequests: exclude --- Engineering and operations work. Customer requests stay in the support Workspace. ``` ## Files in the repository Mirrored files are under `linear/`. Each team has a folder `linear/teams//`, and each project has a folder `linear/projects//`. Below them are folders such as `issues/`, `documents/`, `cycles/`, `labels/`, `users/`, and `customer-requests/`. The paths use stable Linear ids, so a renamed issue or project does not make a second copy. Attachments are in an `assets/` folder next to the Markdown file. Each asset can be up to 25 MiB, and each Linear item up to 100 MiB in total. A file that is too large, not safe, or not available stays a link to Linear. ## Check the setup 1. The Linear workspace shows on **Connectors**. 2. After the merge, the Workspace repository has files under `linear/`. 3. When you change a mirrored issue in Linear, the file of that issue changes. This step tests the Linear webhook. For problems, see [Troubleshooting](/docs/connections#troubleshooting). # Notion connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/notion) The **Notion connector** mirrors selected pages and databases into a [Workspace](/docs/workspaces) repository. ctxpipe reads Notion content. It does not create or change it. On a self-hosted deployment, the operator registers a Notion OAuth app and its webhook first. See [Self-host Notion](/docs/self-hosting/notion). ## What is mirrored A page syncs with all its child pages, and with the child pages that you add later. A database syncs with all its rows. The mirror includes: * pages as Markdown, with headings, lists, tasks, quotes, and code. Links to other mirrored pages become relative links; * each database as an index page, a `table.csv` snapshot, and one Markdown file for each row; * images, files, video, audio, and PDFs, copied into git. Page comments are not mirrored. ctxpipe can list only the pages and databases that are shared with the ctxpipe integration. Share them in the authorization popup, or later in Notion. Each person who can read the Workspace repository can read the mirrored files. ## Connect Notion Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace). For Notion: 1. On **Connectors**, select **Add connection**, then select **Connect** on the **Notion** row. Authorize your Notion workspace, and share the pages and databases that ctxpipe can read. 2. In **Add to Workspaces**, select the Workspaces, then search for the pages and databases. Keep the selection narrow when the Notion workspace has private notes. 3. Merge the pull request that adds `sources/notion/.md`. ## Source file ```markdown --- provider: notion connection: con_example pages: - id: 1f2e3d4c-5b6a-4978-8695-a4b3c2d1e0f9 title: Engineering handbook databases: - id: 0a1b2c3d-4e5f-4061-8728-394a5b6c7d8e title: Runbooks --- ``` ## Files in the repository * Pages: `notion/pages/--/index.md`, with child pages inside the folder of their parent. * Databases: `notion/databases/--/index.md`, `table.csv`, and `rows/--/index.md`. The paths use stable Notion ids, so a renamed page does not make a second copy. Assets are in an `assets/` folder next to their page. Each asset can be up to 25 MiB, and each page or row up to 100 MiB in total. A file that is too large or not safe stays a link to Notion. ctxpipe never writes the download URLs of Notion to git, because they expire. When a page changes in Notion, ctxpipe syncs the whole page tree or database that holds it again. ## Check the setup 1. The Notion workspace shows on **Connectors**. 2. After the merge, the Workspace repository has files under `notion/`. 3. When you change a mirrored page in Notion, the file of that page changes. This step tests the Notion webhook. ## Troubleshooting * **No pages or databases show.** Share them with the ctxpipe integration in Notion, then search again. ctxpipe lists only the pages and databases that you share with the integration. A page that you stop sharing leaves the list. For other problems, see [Troubleshooting](/docs/connections#troubleshooting). # PagerDuty connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/pagerduty) The **PagerDuty connector** mirrors the incidents of selected services into a [Workspace](/docs/workspaces) repository. ctxpipe requests read access and the permission to create its webhook subscription. It does not acknowledge or resolve incidents, and it does not show live on-call state. After you authorize an account, ctxpipe creates one signed webhook subscription for the connection. On a self-hosted deployment, the operator registers a PagerDuty OAuth app first. See [Self-host PagerDuty](/docs/self-hosting/pagerduty). ## What is mirrored ctxpipe syncs the incidents of each selected service from the last 90 days. Each incident is one Markdown file with: * the title, status, urgency, service, assignees, and times; * the body of the alert that started it, with its details, such as a host name; * a short list of the other alerts on the incident; * the incident notes; * image attachments, copied next to the file. ctxpipe removes routing keys and other secret alert fields before it writes to git. If an incident moves to a service that is not selected, ctxpipe deletes its file. ## Connect PagerDuty Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace). For PagerDuty: 1. On **Connectors**, select **Add connection**, then select **Connect** on the **PagerDuty** row. Authorize the app in the PagerDuty popup. 2. In **Add to Workspaces**, select the Workspaces, then the services. If the list is empty, create a service in PagerDuty first. 3. Merge the pull request that adds `sources/pagerduty/.md`. ## Source file ```markdown --- provider: pagerduty connection: con_example services: [PSVC001, PSVC002] --- ``` ## Files in the repository ```text pagerduty/services//incidents/--.md pagerduty/services//incidents/--/assets/… ``` When you remove a service from the scope, the next sync deletes the incident files of that service. ## How incidents stay current PagerDuty sends a signed event when an incident changes, or when a new alert arrives on it. The subscription covers the whole account, but ctxpipe fetches and writes only the incidents of the services in a **Connected** source. It ignores the other events. ## Check the setup 1. The PagerDuty account shows on **Connectors**. 2. After the merge, the Workspace repository has files under `pagerduty/services/`. 3. When you trigger or update an incident of a selected service, its file changes. This step tests the webhook. ## Troubleshooting * **Changes do not arrive.** Make sure that the incident is on a selected service. Then ask the operator to check that the connection still has its webhook subscription, and that `POST /api/v1/webhook/pagerduty` is reachable. * **You reconnect PagerDuty.** ctxpipe keeps or replaces the webhook subscription. It never adds a second one. * **You remove the connector.** ctxpipe also deletes the webhook subscription in PagerDuty. For other problems, see [Troubleshooting](/docs/connections#troubleshooting). # Plain connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/plain) The **Plain connector** writes Help Center articles and support threads from [Plain](https://www.plain.com) into a [Workspace's](/docs/workspaces) GitHub repository. ctxpipe then indexes them like other knowledge files. Agents can read your support documentation and see what customers report. A source file in the Workspace holds the scope. A pull request adds it, so a person reviews the scope before content syncs. ctxpipe only reads from Plain. It does not change threads, articles, or customers. The one write is a webhook target for thread activity, which ctxpipe adds and removes. ## What is mirrored You select Help Centers, thread labels, or both, for each Workspace. * **Help Center:** one file for each published article, with its group and its body as Markdown. Images are text stubs in this version. * **Threads:** one file for each of the 100 newest threads with a label. A thread file has the ref, status, priority, labels, the customer name, and the first 50 messages in order, each as a quote. Internal notes are marked. A longer thread says that Plain has more messages. ctxpipe writes people by name only. In thread text, ctxpipe removes email addresses, phone numbers in common formats, and links that carry a credential. ctxpipe keeps only `https:` links. A link with text keeps the text and loses the link. A bare URL that is not `https:` becomes `[link removed]`. Other personal data in the text can remain. Thread messages are text from your customers, and they can hold personal data. Git keeps every version of a file, so a removed thread stays in the history of the repository. Select labels that hold the threads your Workspace needs, and keep the repository private. ## Prerequisites * A Plain admin who can add a machine user and an API key. * A [Workspace](/docs/workspaces) whose repository the ctxpipe GitHub App can write to. The repository must use format 2 or higher. If it does not, the dialog shows **Needs the format upgrade first.** next to the Workspace. See [Upgrades](/docs/workspaces/settings#upgrades). * Permission to merge the pull request that adds the source. ## Where the scope lives The connection holds the API key only. The scope is in a source file in each Workspace, at `sources/plain/.md`. The front matter holds the Help Center ids and the label type ids. The body is an optional description of what the source is for. ```markdown --- provider: plain connection: con_example helpCenters: [hc_01EXAMPLE] labels: [lt_01EXAMPLE] --- Product documentation and bug reports for the support Workspace. ``` The scope goes live after the merge. See [Workspace settings](/docs/workspaces/settings#sources) for draft scope and live scope. A Workspace can have many Plain source files, and each Workspace has its own scope. ## Setup flow ### Create a machine user and an API key In Plain, open **Settings → Machine Users** and add a machine user for ctxpipe. Add an API key with only these permissions: `helpCenter:read`, `helpCenterArticle:read`, `helpCenterArticleGroup:read`, `labelType:read`, `label:read`, `thread:read`, `timeline:read`, `customer:read`, `user:read`, `machineUser:read`, `webhookTarget:create`, and `webhookTarget:delete`. Copy the key. Plain shows it one time. ### Connect Plain In ctxpipe, open **Connectors**, select **Add connection**, then select **Connect** on the **Plain** row. Paste the key and select **Connect Plain**. ctxpipe checks the key and its permissions, keeps it encrypted, and adds one webhook target for thread activity. ### Add to Workspaces The next step is **Add to Workspaces**. Select the Workspaces that sync from Plain. Then select the Help Centers and labels. When you select more than one Workspace, choose **Same scope for each Workspace** or **A different scope for each Workspace**. Choose **Add to Workspaces**. You can add the connection to a Workspace later. Expand the connector on **Connectors**. When no Workspace uses the connection, choose **Add to a Workspace**. Otherwise, choose **Manage**. ### Merge the pull requests ctxpipe opens one pull request for each Workspace. Each pull request adds `sources/plain/.md`. The dialog shows **Merge the pull request** (or **Merge the pull requests**) with a link to each one. Review the selected Help Centers and labels, and the repository audience, then merge. After each merge, the dialog shows **Pull request merged**. The source does not sync before the merge. ### Wait for the first sync After GitHub reports the merge, ctxpipe reads the source file and writes the first Plain files. The Workspace indexes the new files after that. See [How indexing works](/docs/workspaces/indexing) for the lifecycle. ## Change what syncs Edit, rename, or remove a source in the **Settings** of the Workspace. See [Workspace settings](/docs/workspaces/settings#sources). When you remove a Help Center or a label from the scope, ctxpipe deletes its files in a sync commit. To change the Workspaces of the connection, use **Manage** on the connector card. **Manage** opens the same **Add to Workspaces** step. It starts from the current Workspaces and scopes. ## Files in the repository ```text plain/help-centers//--.md plain/threads/labels//.md ``` The source owns these folders. Do not edit the files. ctxpipe writes them again on each sync. When an article is unpublished, or a thread leaves the newest 100 of its label, ctxpipe deletes its file in the same commit. ## Keeping content current * **Thread webhook:** a new message, a note, or a label change on a thread starts a sync of the labels at the end of a 10-minute window. More events in the same window start no more syncs. * **Daily sync:** ctxpipe syncs each connection once a day. Help Center articles use this sync, because Plain sends no article events. * **Sync now:** an owner or admin can select **Sync now** in the menu of the connector card. ## Verify the complete flow 1. Connect Plain and confirm that the name of the Plain workspace appears on **Connectors**. 2. Choose **Add to Workspaces**, select a Workspace, and select at least one Help Center or label. 3. Merge the pull request that adds `sources/plain/.md`. 4. Confirm that the Workspace repository has files under `plain/`. 5. Add a note to a real thread with a selected label, not a Plain test thread. Confirm that the thread file changes after the 10-minute window. 6. Confirm that the Workspace index is ready in its **Settings**. Steps 3 and 4 test the GitHub push webhook and the first sync. Step 5 tests the Plain webhook delivery. ## Troubleshooting ### A test thread does not sync Plain shows a test thread only to the person who created it, so the API key cannot read it. To test the thread sync, send an email to your support address, and add a selected label to the new thread. ### A label or a Help Center does not sync Plain refuses a read when the key does not have a permission in the list above. The source can still show OK. Add that permission to the key in Plain, then select **Sync now** in the menu of the connector card. ### The setup names missing permissions The key does not have every permission in the list. Create a new key on the machine user with all of them, and paste it. ### Add to a Workspace shows an error The dialog shows a message about the format, the write access, or your role. See [Troubleshooting](/docs/connections#troubleshooting). ### No pull request appears Confirm that the GitHub App has write access to the Workspace repository. If the dialog reports an error, close it and try again. ### The source shows a status other than OK Read the status and the reason of the row in **Settings**. See [Source status](/docs/workspaces/settings#source-status). When a selected Help Center or label is deleted or archived in Plain, ctxpipe tries it in three sync cycles and keeps its files. Then the source shows **Out of access**. Remove the Help Center or the label from the source. ### The card says that threads sync once a day Plain did not accept the webhook target. Plain sends webhooks only to a public HTTPS address, and a workspace can have at most 25 targets. Select **Reconnect Plain** on the card and paste the key again to try again, or use the daily sync and **Sync now**. A reconnect replaces the target, so it also helps when someone deleted the target in Plain. ### The card shows “Needs reconnect” Plain refused the saved key, for example because someone deleted it. ctxpipe keeps the files and stops the sync. Select **Reconnect Plain** on the card and paste a new key. A permission that the key does not have does not show here. See [A label or a Help Center does not sync](#a-label-or-a-help-center-does-not-sync). ### Removing the connector Choose **Remove connector** in the menu of the connector card. See [How content stays current](/docs/connections#how-content-stays-current) for what stays in Git. ctxpipe also deletes the webhook target in Plain while the key works. When the key no longer works, delete the target in **Settings → Webhooks** in Plain. ### More than one ctxpipe target in Plain A failed request can leave an extra ctxpipe webhook target in Plain. ctxpipe refuses its events. Delete each ctxpipe target in **Settings → Webhooks** in Plain except the newest, or delete all of them and select **Reconnect Plain**. # Slack connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/slack) The **Slack connector** keeps the Slack threads that matter, such as decisions, incidents, and design debates, in a [Workspace](/docs/workspaces). It does not mirror channels or read Slack in the background. A person mentions the bot in a thread, and ctxpipe saves that thread as Markdown. The pull request that adds the channels to a Workspace is the reviewed part. Each capture after that is a commit, so git history records who captured which thread, and when. A Slack workspace can connect to one ctxpipe organization. On a self-hosted deployment, the operator creates the Slack app first. See [Self-host Slack](/docs/self-hosting/slack). ## Slack connector or Slack MCP | Tool | Use it to | | ------------------------------------------------------------------ | ------------------------------------------------------- | | **ctxpipe Slack connector** (this page) | Keep chosen threads in git, where ctxpipe indexes them. | | **[Slack MCP server](https://docs.slack.dev/ai/slack-mcp-server)** | Search and read live Slack from your coding agent. | The two do different jobs. You can use both. ## Connect Slack Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace). For Slack: 1. On **Connectors**, select **Add connection**, then select **Connect** on the **Slack** row. Authorize the Slack workspace. 2. In **Add to Workspaces**, select the Workspaces, then the channels for each one. When captures can include private channels, use a separate Workspace with a private repository. 3. Merge the pull request that adds `sources/slack/.md`. Captures start after the merge. ## Capture a thread 1. Invite the bot to the channel with `/invite`. The connector card shows the **@handle** of the bot. 2. In the thread, mention the bot: ```text @your-bot-handle capture this ``` A mention with no other text also captures. A mention outside a thread does not capture. ctxpipe replies in the thread with **ctxpipe agent working…**, then changes the reply to **Engineering context captured into**, with a link to the file in each Workspace that has the channel. ctxpipe does not answer other questions in Slack. It replies with what it can do. Later replies in the thread are not captured by themselves. Mention the bot again to capture the thread again. The new capture replaces the file. ## What is captured ctxpipe saves the thread in each Workspace whose **Connected** Slack source has the channel, with one commit for each Workspace: * the file is `slack/channels//threads////thread.md`; * the front matter holds the capture time, the Slack link, the channel and thread ids, and the handle and name of the person who captured it; * the authors and the people mentioned show by name; * files and embedded media are copied into the `assets/` folder of the thread. Each asset can be up to 25 MiB, and each thread up to 100 MiB in total; * a thread longer than 500 messages keeps its oldest 500, and the reply in Slack says so. Not included: channel mirrors, history backfill, direct messages, and automatic updates after a capture. When you uninstall Slack, or remove a channel from a source, the captured threads stay in the repository. ## Troubleshooting * **A mention does nothing.** Check that the connector shows **Connected**, that a **Connected** source has the channel, and that the bot is in the channel. Use Slack autocomplete to mention the bot. If no Workspace has the channel, the bot replies "Nothing captured: no Workspace has this channel." * **A private channel does not work.** Invite the bot to the channel first. * **The source is OK, but the bot is not in the channel.** The reason on the row tells you to invite the bot. For other problems, see [Troubleshooting](/docs/connections#troubleshooting).