# Self-host ctxpipe (https://docs.ctxpipe.ai/docs/self-hosting)
Self-hosted ctxpipe is the same product as hosted ctxpipe. It runs in
infrastructure that you control. The ctxpipe team is not in the data path. Your
deployment holds the credentials and the source data. Model requests go from
your deployment to the provider that you choose.
## Choose a deployment path
| Path | Use it when | You operate |
| ------------------------------------------------------ | --------------------------------------------------------------------- | --------------------------------------------------- |
| [Docker Compose](/docs/self-hosting/deployment/docker) | You evaluate ctxpipe, or run a small internal deployment on one host. | One Compose project and its volumes |
| [AWS CDK](/docs/self-hosting/deployment/aws) | You run ctxpipe in your own AWS account. | The `CtxPipe` construct, a domain, and model access |
Each path provides the same services in a different way:
| | Docker Compose | AWS CDK |
| --------------- | -------------------------------------- | ------------------------------------------- |
| Postgres | `pgvector/pgvector:pg17` container | Aurora PostgreSQL 16 |
| Graph database | FalkorDB container | Amazon Neptune |
| Knowledge store | ClickHouse container | ClickHouse on one EC2 host, old parts on S3 |
| Sandboxes | Docker-in-Docker (`dind`) | A Graviton EC2 Docker host |
| Public origin | Your TLS proxy in front of the backend | Route 53 and a load balancer |
Start with the [quickstart](/docs/self-hosting/quickstart). It covers each path.
## What you run
* **Backend**: auth, the REST API, the MCP endpoint, webhooks, and the public
origin. It also serves the UI.
* **UI**: the web app, behind the backend.
* **Worker**: the background jobs for ingestion, connectors, and Workspace
writes.
* **Codesearch**: repository clones and code search indexes, on persistent
storage.
* **PostgreSQL with pgvector**: the application data and the job state.
* **A graph database**: FalkorDB by default.
* **ClickHouse**: the new knowledge store for the documents, their sections,
and their relations. See
[Knowledge store](/docs/self-hosting/configuration#knowledge-store).
* **A sandbox host**: runs Workspace chat and ingestion in isolated containers.
Without one, chat answers `503`. For another platform, see
[Chat sandboxes](/docs/self-hosting/configuration#chat-sandboxes).
OpenTelemetry export is off by default. See
[Observability](/docs/self-hosting/configuration#observability).
## What you own
* One stable public HTTPS origin, `AUTH_BASE_URL`, for the browser, the API,
auth, MCP, and the provider callbacks.
* Durable storage, backups, and recovery for Postgres, the graph database,
ClickHouse, and codesearch. See [Graph databases](/docs/self-hosting/graph-databases).
* The secrets for auth and encryption, and the model provider credentials for
chat, ingestion, and embeddings. See [Models](/docs/self-hosting/models).
* A provider app for each connector that you turn on: a GitHub App for
repositories, and OAuth apps for Atlassian, Slack, Linear, Notion, and
PagerDuty. See the source connector pages in this section.
* Monitoring, migrations, upgrades, and rollback.
After the deployment is healthy, people use it like hosted ctxpipe. Send them
to [Connections](/docs/connections) and
[Connect your agent](/docs/mcp/mcp-docs).
# Connections (https://docs.ctxpipe.ai/docs/connections)
A **connection** gives ctxpipe access to one of your tools. A **source** tells
one [Workspace](/docs/workspaces) what to read through a connection. ctxpipe
writes the content into the Workspace repository as Markdown files, so you can
read, review, and keep each file that ctxpipe and your agents use.
* A **connection** holds only the authorization: the OAuth grant, API access
token, or GitHub App installation, and its webhooks. An organization owner or
admin adds it one time for the organization, on **Connectors**.
* A **source** is one Markdown file in a Workspace, at
`sources//.md`. It names the connection, the scope (for
example Linear teams or Notion pages), and an optional description.
One connection can feed many Workspaces. Each Workspace has its own source file
and its own scope.
## Available connectors
| Connector | Type | What enters the Workspace | Scope |
| ------------------------------------------------------------------------ | ------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------- |
| [GitHub](/docs/connections/source-connectors/github) | Git source | Linked repositories, indexed for code search and extraction, and their issues and closed pull requests | One repository, and an optional `ref` |
| [Linear](/docs/connections/source-connectors/linear) | Scoped mirror | Issues, projects, cycles, and related content | Teams and projects |
| [Notion](/docs/connections/source-connectors/notion) | Scoped mirror | Pages and databases | Pages and databases |
| [Confluence](/docs/connections/source-connectors/confluence) | Scoped mirror | Spaces and pages | Spaces, with all pages or a page list |
| [PagerDuty](/docs/connections/source-connectors/pagerduty) | Scoped mirror | Incidents of the selected services | Services |
| [Buildkite](/docs/connections/source-connectors/buildkite) | Scoped mirror | The latest 25 finished builds of the selected pipelines | Pipelines |
| [Plain](/docs/connections/source-connectors/plain) | Scoped mirror | Help Center articles and the newest threads of selected labels | Help Centers and thread labels |
| [Slack](/docs/connections/source-connectors/slack) | Capture on request | One thread at a time, when someone mentions the bot | Channels |
| [Plain git](/docs/workspaces/linked-repositories#plain-git-repositories) | Git source | A public repository on any git host, with no connection | A URL |
ctxpipe indexes the docs in linked repositories with the code: README files,
`docs/**`, ADRs, `AGENTS.md`, and skills.
The main difference is the scope. A scoped mirror keeps an approved
subset current. Slack has no history backfill and no channel mirror: a person
captures one thread when the team needs it later.
## Connect GitHub first
The ctxpipe GitHub App lets ctxpipe read and write Workspace repositories,
index linked code repositories, and open the pull requests for source changes,
upgrades, and Workspace chat. The other connectors write into a Workspace
repository on GitHub, so they need it too. See the
[GitHub connector](/docs/connections/source-connectors/github).
## Add a tool to a Workspace
Before you start, make sure that:
* you are an organization owner or admin;
* the ctxpipe GitHub App can write to the Workspace repository, and the
repository uses format 2 or higher (see
[Upgrades](/docs/workspaces/settings#upgrades));
* you can merge pull requests in the Workspace repository.
### Connect the tool
Open **Connectors**, choose **Add connection**, then choose **Connect** on
the row of the tool. To find a tool, type its name in the search field or
select its group. A tool that is not available yet has **Ask for it**.
Follow the authorization steps for the tool. Buildkite uses an API access
token. The connector page has the details for each tool.
### Add it to Workspaces
The next step is **Add to Workspaces**. Select the Workspaces, then select
the scope. For more than one Workspace, choose **Same scope for each Workspace** or **A different scope
for each Workspace**. Then choose **Add to Workspaces**.
You can do this later. Expand the connector on **Connectors** and choose
**Add to a Workspace** (or **Manage**), or use **Add** under **Sources** in
the **Settings** of a Workspace.
### Merge the pull requests
ctxpipe opens one pull request for each Workspace. Each pull request adds the
source file. Review the scope and who can read the repository, then merge.
The source does not sync before the merge.
### Wait for the first sync
After the merge, ctxpipe makes the first mirror. The Workspace then indexes
the new files like any other knowledge file.
To change what a Workspace reads, edit, rename, or remove its source. See
[Workspace settings](/docs/workspaces/settings#sources).
## How content stays current
* **Webhooks.** Linear, Notion, Confluence, PagerDuty, and Plain tell ctxpipe
when content changes. ctxpipe updates the files of the sources whose scope
includes the change. Events that arrive before a source is live are skipped.
* **Buildkite webhooks.** A finished build starts a full sync after a 10-minute
window. More events in that window share the same sync.
* **Daily full sync.** Once a day, ctxpipe syncs each connection that has a
source with the status **Connected**. This sync finds content that the provider
deleted, and content that a webhook missed.
* **Sync now.** The menu of the connector on **Connectors** has **Sync now** for
Confluence, Linear, Notion, PagerDuty, Buildkite, and Plain. It starts the
same full sync at once. Only an owner or admin sees it.
Do not edit mirrored files by hand. The next sync overwrites them. When you
remove a source, the next sync deletes its files in one commit. Git history
keeps them.
**Remove connector**, in the same menu, removes the authorization. The sources
of the connection stop syncing. The source files and the files that ctxpipe
committed stay in git.
## Git is the review boundary
A source file changes only through a pull request that a person merges. The
pull request is the review of the scope. No agent can add or change a source
without that review.
* **Scoped mirrors** write Markdown under the folder of the provider, for
example `linear/` or `notion/`, in each Workspace repository.
* **Slack** captures commit directly to the default branch of each Workspace
that has the channel. The mention in the thread is the approval, and git
history records it.
Each person who can read the Workspace repository can read its mirrored
content. Use separate Workspaces when different sources need different
audiences.
## Scope limits
Each source has a maximum number of items, so that one sync stays inside the
API rate limits of the provider. For more items, add a second source with the
same connection.
| Source | Maximum in one source |
| -------------- | --------------------------------------------------------------------------- |
| **Confluence** | 100 spaces, and 500 selected pages in one space (select all pages for more) |
| **Slack** | 100 channels |
| **Linear** | 50 teams and 100 projects |
| **Notion** | 100 root pages and 100 databases |
| **PagerDuty** | 100 services |
| **Buildkite** | 50 pipelines |
| **Plain** | 50 Help Centers and 50 thread labels |
A GitHub source holds one repository, so it has no limit.
## Troubleshooting
| Problem | What to do |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Authorization fails | Allow popups and try again. On a self-hosted deployment, ask the operator to check the OAuth app of the provider and its callback URL. |
| **Add to Workspaces** shows an error | The message names the cause: **Needs the format upgrade first.** next to a Workspace (see [Upgrades](/docs/workspaces/settings#upgrades)), **This Workspace is read-only** (see [Write access](/docs/workspaces/settings#write-access)), or **Ask an admin to choose what this source syncs** (your role). |
| No pull request appears | Make sure that the GitHub App can write to the Workspace repository. Close the dialog and try again. |
| The source status is not **Connected** | Read the status and the reason in **Settings**. See [Source status](/docs/workspaces/settings#source-status). |
| Changes in the tool do not arrive | Choose **Sync now**. If changes still do not arrive, ask the operator to check the webhook of the provider. |
| A sync fails | Choose **Sync now**. If it fails again, ask the operator to check the worker logs, the GitHub write access, and outbound access to the API of the provider. |
| The authorization was revoked | Reconnect the tool from its connector card. Updates stop until you do. |
## More connectors
We plan more connectors for engineering systems, for example Jira and Sentry.
On a self-hosted deployment, the operator must configure the OAuth app and the
webhooks of a provider before its connector works. See the source connector
pages under [Self-hosting](/docs/self-hosting). Buildkite uses a token for each
connection. See [Self-host Buildkite](/docs/self-hosting/buildkite).
## Agent access is separate
MCP clients are not connections. Claude Code, Cursor, CI agents, and other MCP
clients connect to ctxpipe through the MCP endpoint. See
[Connect your agent](/docs/mcp/mcp-docs).
# Common workflows (https://docs.ctxpipe.ai/docs/getting-started/common-workflows)
Each workflow starts from the same Workspace. People use
[Workspace chat](/docs/workspaces/chat). Coding agents use the
[MCP tools](/docs/mcp/use-from-your-agent).
## Plan a change across repositories
Link the repositories that a team owns to its Workspace. Before an agent plans
a change, it searches the knowledge for the decisions and standards that apply,
then searches the code in each linked repository. The
[knowledge graph](/docs/workspaces/graph) shows which services, APIs, and
libraries the change touches.
```text
Use ctxpipe before you plan this change. Find the decisions and standards that
apply, and the code in other repositories that calls this API.
```
## Give every agent the same instructions
Keep `AGENTS.md`, skills, ADRs, and contribution guides in linked repositories
or in the Workspace repository. ctxpipe returns them with the code that they
govern. Claude Code, Cursor, Codex, and your review bots then follow the same
guidance as your engineers. One MCP endpoint serves each client, so you do not
build retrieval for each agent.
## Bring product and incident context into git
Linear, Notion, Confluence, PagerDuty, Buildkite, and Plain mirror the scope
that you approve into the Workspace. Slack captures a thread when someone mentions the bot. The
result is Markdown with a git history, not a hidden index. An agent can cite
the issue or the incident behind a decision. See
[Connections](/docs/connections).
## Write it down once
When chat or an agent finds a fact that the team needs later, ask Workspace
chat to write it as a knowledge file. Review the diff and open a pull request. The next
person and the next agent start from that file.
[Local memory](/docs/mcp/local-memory) does the same for the lessons that
agents learn in one repository.
## Run headless agents and review bots
CI jobs, review bots, and other agents that cannot sign in through a browser
use an organization API key. See [API keys](/docs/mcp/api-keys).
## Run ctxpipe on your own infrastructure
The open-source core runs on Docker Compose, Terraform, or AWS CDK. Your source
data and model calls stay in your infrastructure. See
[Self-hosting](/docs/self-hosting).
# Get support (https://docs.ctxpipe.ai/docs/getting-started/get-in-touch)
## Product and deployment questions
* [Book a 30-minute call](https://cal.com/ctxpipe/30min) for product,
deployment, or commercial questions.
* [Request hosted early access](https://ctxpipe.ai/early-access).
* Visit [ctxpipe.ai](https://ctxpipe.ai) for the product overview.
## Community and technical support
* Join the [ctxpipe Discord](https://ctxpipe.ai/discord) for questions and
discussion.
* Report reproducible bugs or request features through
[GitHub issues](https://github.com/ctxpipe-ai/ctxpipe/issues).
* Follow [@ctxpipe on X](https://x.com/ctxpipe) for product updates.
## Documentation and source
* [Documentation](https://docs.ctxpipe.ai)
* [Source code](https://github.com/ctxpipe-ai/ctxpipe)
* [Open-source overview](/docs/resources/open-source)
# Getting started with ctxpipe (https://docs.ctxpipe.ai/docs/getting-started)
ctxpipe is context management for software factories. A software factory is the
set of coding agents, and the harnesses that run them, that a team uses to
write software.
Your coding agents can reach every repository, ticket, doc, and Slack thread.
They cannot tell which parts are true. ctxpipe gives them one organized,
current, and inspectable source of context: a git repository of Markdown that
you own, and that you can read, review, and correct.
1. **Connect.** [Link your code repositories](/docs/workspaces/linked-repositories),
and [select what to bring in](/docs/connections) from Linear, Notion,
Confluence, PagerDuty, Buildkite, Plain, and Slack. ctxpipe writes it as
Markdown into one git repository for each [Workspace](/docs/workspaces).
2. **Keep current.** Background jobs mirror changes from your tools, and read a
linked repository again after each push. Each job makes
[one small commit](/docs/workspaces#what-ctxpipe-writes) that you can review
or revert. Then ctxpipe
[indexes the new commit](/docs/workspaces/indexing) for search and the graph.
3. **Serve.** Claude Code, Cursor, Codex, OpenCode, and VS Code read the same
Workspace [through one MCP endpoint](/docs/mcp/mcp-docs). ctxpipe returns
sources, not answers. Your agent reasons with its own model.
## Who ctxpipe is for
ctxpipe is for engineering teams that run coding agents across many teams and
repositories, and for the platform engineers who build the agent harness. It
helps when:
* agents repeat mistakes that a decision record or a runbook already explains;
* the context for one change is in several repositories and tools;
* you must see what an agent read, and correct it in one place.
For the full problem, the teams that ctxpipe is not for, and how ctxpipe
compares with other approaches, see [ctxpipe.ai](https://www.ctxpipe.ai).
## Key terms
| Term | Meaning |
| ------------------------ | ----------------------------------------------------------------------------------- |
| **Organization** | Your team in ctxpipe. It holds members, connections, and Workspaces. |
| **Workspace** | One body of knowledge, for example one product or one team. |
| **Workspace repository** | The git repository of Markdown that holds the knowledge of a Workspace. You own it. |
| **Linked repository** | A code repository that a Workspace indexes for search. |
| **Connection** | The authorization for one tool, such as Linear or Notion. |
| **Source** | A file in the Workspace repository that says what to read through a connection. |
## Choose your setup
## Give these docs to your agent
Point your coding agent at
[docs.ctxpipe.ai/llms.txt](https://docs.ctxpipe.ai/llms.txt). It lists each
page as Markdown. For all the pages in one file, use
[llms-full.txt](https://docs.ctxpipe.ai/llms-full.txt). To give the agent one
page, choose **Copy Markdown** under the title of that page.
## Learn more
# Hosted quickstart (https://docs.ctxpipe.ai/docs/getting-started/quickstart)
This guide sets up hosted ctxpipe with one Workspace, and connects it to your
coding agent. The agent connection is the goal. Workspace chat is a quick check
that the Workspace has the right knowledge.
We help you set up your first Workspace.
[Request early access](https://ctxpipe.ai/early-access) or
[book a demo](https://cal.com/ctxpipe/30min). To run ctxpipe on your own
infrastructure, start with the
[self-hosting quickstart](/docs/self-hosting/quickstart).
## Before you start
You need:
* permission to install the ctxpipe GitHub App on your GitHub account or
organization, or a GitHub owner who can approve it;
* a GitHub repository for the knowledge of the Workspace. An empty repository
is good. You can create one during setup;
* the code repositories to link;
* an MCP client: Claude Code, Cursor, Codex, OpenCode, or VS Code with
Copilot.
## 1. Sign up and create your organization
Sign up at [app.ctxpipe.ai](https://app.ctxpipe.ai), then create an
organization. With a work email, ctxpipe suggests a name from your email
domain. If a teammate invited you, you join their organization.
**Home** then shows the **Get started** guide. It lists the next steps and marks
each step done when ctxpipe sees the result.
## 2. Connect GitHub
In the guide, choose **Connect GitHub**. Install the ctxpipe GitHub App on the
account or organization that owns your repositories. Give it access to the
repository for the knowledge, and to the code repositories that you want to
link. You can add repositories later in GitHub.
A GitHub account can belong to one ctxpipe user only. See
[GitHub connector](/docs/connections/source-connectors/github).
## 3. Create a Workspace
Choose **Create Workspace**, then choose the Workspace repository:
* **Select GitHub**: use a repository that you have on GitHub.
* **Create on GitHub**: create a new repository on GitHub, then select it.
* **Paste URL**: use any other git URL. ctxpipe cannot write to it, so the
Workspace is read-only.
ctxpipe adds `AGENTS.md` and a knowledge skill if they are missing, and starts
to read the repository. See [Create a Workspace](/docs/workspaces/create-workspace).
## 4. Link code repositories
The next step shows the **Sources** of the new Workspace. Choose **Add
repositories**, select the code repositories, and choose **Open a pull
request**. Merge the pull request on GitHub. Each repository then goes from
**Pending** to **Indexing** to **Indexed**.
To link more repositories later, open the **Settings** tab of the Workspace.
See [Linked repositories](/docs/workspaces/linked-repositories).
## 5. Ask a question
Type a question in the Workspace, and ask for the files behind the answer:
```text
What should a new engineer read first, and which files support that answer?
```
You can ask while ctxpipe indexes. The answers improve when indexing is done.
See [Workspace chat](/docs/workspaces/chat).
## 6. Connect your coding agent
The **Get started** guide has a **Use ctxpipe from your agents** block with the
setup command. Copy it, and run it in a repository:
```bash
npx ctxpipe init
```
The CLI signs you in, asks which organization and agent clients to configure,
and writes the MCP configuration. The first time your agent connects, it asks
you to authorize ctxpipe in the browser. See
[Connect your agent](/docs/mcp/mcp-docs).
Then give the agent a task:
```text
Use ctxpipe before you plan this change. Search the knowledge for the
standards and decisions that apply, search the code for the related areas,
and say what still needs a check in the code.
```
## 7. Connect your tools and invite your team
* Open **Connectors** to add Linear, Notion, Confluence, PagerDuty, Buildkite,
Plain, or Slack. See [Connections](/docs/connections).
* Open **Organization settings → Members** to invite your team. See
[Organizations and members](/docs/managing-organization).
## Setup is complete when
* the linked repositories show **Indexed** in the **Settings** of the Workspace;
* Workspace chat answers a question and names useful files;
* your agent can call `list_workspaces` and `search_knowledge`.
# Organizations and members (https://docs.ctxpipe.ai/docs/managing-organization)
An **organization** is the access boundary for Workspaces, connections,
conversations, and MCP. A person can belong to more than one organization, and
switch between them with the organization switcher.
* Each organization has a name and a unique URL slug.
* Workspace and connector pages always work inside the organization in the URL.
* Each member of an organization can open each of its
[Workspaces](/docs/workspaces).
## Invite by email
Open **Organization settings → Members** and send an invitation. The invited
person gets an email with a link.
* If they have an account, the link adds them to the organization with the
role of the invitation.
* If they do not, they sign up first with the same email. ctxpipe then applies
the invitation.
## Roles
Each member has one role in each organization:
| Role | What the role can do |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| **owner** | All that an admin can do. Also delete the organization. |
| **admin** | Manage members and invitations. Add and remove connections, and start a sync. Start Workspace upgrades. Create organization API keys. |
| **member** | Open and use the Workspaces of the organization, and chat in them. See the status of connections and upgrades. |
Admin actions need a signed-in browser session. An API key or an MCP token
cannot do them, even when its owner is an admin.
## Access follows membership and source scope
Membership gives a person access to the organization in ctxpipe. It does not
give access to more content in your tools:
* the GitHub App installation controls which code ctxpipe can read;
* the scope of each source controls which Linear, Notion, Confluence,
PagerDuty, Buildkite, or Plain content ctxpipe mirrors;
* the permissions of the Workspace repository control who can read the
mirrored content in git;
* Slack captures need the bot in the channel.
Use roles to control administration in ctxpipe. Keep the permissions of your
tools, and the audience of each Workspace repository, as narrow as you need.
An MCP client reads only the organizations that you selected for it, and only
while you are a member. See [Connected apps](/docs/mcp/connected-apps).
# API keys (https://docs.ctxpipe.ai/docs/mcp/api-keys)
OAuth is the default for a person at a laptop. Use an API key only when a
client cannot finish the browser sign-in: a CI job, a headless agent, a review
bot, or a host that accepts only a fixed header.
## Choose a key
| | Organization key | Personal key |
| ------------------ | ------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Where to create it | **Organization settings → API Keys** | **User account → API Keys** |
| Who can create it | An organization owner or admin | Each user, for themselves |
| Acts as | The organization, not the person who created it | You, as a member |
| Reads | Its one organization. The `org` argument must name it. | Each organization that you are a member of. ctxpipe checks your membership on each call. |
| Works on | `/mcp` only | `/mcp`, and the REST API as you |
Use an organization key for shared automation. It continues to work when the
person who created it leaves the organization.
A new key expires after 30 days by default, and allows 1,000 requests per hour.
You can create a key that never expires, but treat it as a long-lived secret.
ctxpipe shows the secret one time only. Give each key a name.
To rotate a key, create a new key, change `CTXPIPE_API_KEY` to it, then revoke
the old key.
## Send the key
Send the key in the `x-api-key` header to `/mcp`. If a host can send only a
Bearer token, send `Authorization: Bearer `. A Bearer API key works only
on `/mcp`, not on the REST API.
Keep the secret out of files. Put a reference to the `CTXPIPE_API_KEY`
environment variable in the client configuration. The reference is safe to
commit.
The CLI writes that reference for you:
```bash
npx ctxpipe mcp add --client cursor --scope both --auth api-key --non-interactive
```
The CLI never creates a key and never reads `CTXPIPE_API_KEY`.
Start from the client configuration in
[Configure a client by hand](/docs/mcp/mcp-docs#configure-a-client-by-hand),
and add the header entry for your client:
| Client | File | Header entry |
| ----------------- | ----------------------------------------------------- | --------------------------------------------------------------------------- |
| Claude Code | `.mcp.json` | `"headers": { "x-api-key": "${CTXPIPE_API_KEY}" }` |
| Cursor | `.cursor/mcp.json` or `~/.cursor/mcp.json` | `"headers": { "x-api-key": "${env:CTXPIPE_API_KEY}" }` |
| Codex | `.codex/config.toml` or `~/.codex/config.toml` | `env_http_headers = { "x-api-key" = "CTXPIPE_API_KEY" }` |
| OpenCode | `opencode.json` or `~/.config/opencode/opencode.json` | `"headers": { "x-api-key": "{env:CTXPIPE_API_KEY}" }`, and `"oauth": false` |
| VS Code / Copilot | `.vscode/mcp.json` | `"headers": { "x-api-key": "${env:CTXPIPE_API_KEY}" }` |
Each client uses its own syntax for the variable. Codex needs the name of the
variable, not `$VAR` and not the secret. In OpenCode, `"oauth": false` stops
the client from opening the browser after a `401`.
For example, a Claude Code `.mcp.json` with an API key:
```json
{
"mcpServers": {
"ctxpipe": {
"type": "http",
"url": "https://app.ctxpipe.ai/mcp",
"headers": {
"x-api-key": "${CTXPIPE_API_KEY}"
}
}
}
}
```
## Set `CTXPIPE_API_KEY`
The process of the MCP client must have the variable. An `export` in the
terminal where you ran the CLI is not enough for an app that you start from
the Dock or the Start menu.
* **CLI agents** (Claude Code, Codex, OpenCode, Cursor CLI): export the
variable in the terminal, then start the agent from that terminal. To set it
for each session, put it in `~/.zshenv` or `~/.bash_profile`. The Cursor CLI
can start a shell that does not read `.zshrc`.
* **Desktop apps** (Cursor, VS Code): set the variable for your user session,
then quit and open the app again. On macOS, run
`launchctl setenv CTXPIPE_API_KEY ''`. On Linux, use `environment.d`.
On Windows, use the user environment variables. You can also start the app
from a terminal that has the variable, for example `cursor .`.
* **CI**: store the key as a secret of your CI system, and expose it to the job
as `CTXPIPE_API_KEY`.
To test the key, run:
```bash
CTXPIPE_API_KEY= npx ctxpipe doctor mcp --url "https://app.ctxpipe.ai/mcp"
```
The status `ready-for-api-key` means that the key works.
## Troubleshooting
* **An organization key gets `401` on the REST API.** This is correct.
Organization keys work only on `/mcp`. Use a personal key or a browser
session for the REST API.
* **A tool gets `org_not_found`.** An organization key reads only its own
organization. Make the `org` argument match it.
* **The client sends no key.** A variable in the environment is not enough by
itself. The client configuration must map it to the `x-api-key` header.
# Connected apps (https://docs.ctxpipe.ai/docs/mcp/connected-apps)
When an MCP client signs in to ctxpipe with OAuth, you approve it on a consent
page. **Connected apps** lists each app that you approved. Use it to change
what an app can read. You do not need to reconnect.
## Open the page
Open your account settings and choose **Connected apps**. On hosted ctxpipe,
the page is at `https://app.ctxpipe.ai/.auth/account/connected-apps`. A
self-hosted deployment uses the same path on its own origin.
For each app, the page shows the client name, what it can do, and the
organizations that it can read. **Read** is the normal access for an MCP client.
**Changes** shows only when you also ticked **Also allow changes** on the
consent page.
## Choose the organizations
On the consent page, ctxpipe lists your organizations with check boxes. The
organization of your current session is selected. Select at least one.
An app can read only the organizations in its set. When you join a new
organization, ctxpipe does not add it to the apps that you approved before.
To change the set:
1. Choose **Edit organizations** for the app.
2. Select or clear organizations.
3. Save.
The change applies on the next tool call. The app does not need to reconnect.
## Revoke an app
Choose **Revoke** for the app. The app loses access at once, and it must ask you
to authorize it again.
An older connection that holds one organization shows **Reconnect to choose
more organizations**. You can only revoke it.
## When a tool says `org_not_found`
The tool call named an organization that the connection cannot read. One of
these is true:
* the organization is not in the set of the app. Add it with **Edit
organizations**, then call the tool again;
* you are no longer a member of the organization;
* the slug is wrong. Call `list_workspaces` with no `org` to see the
organizations of the connection.
An organization API key reads only its own organization. Any other `org` gives
the same error.
## API keys
API keys are not apps, so they are not on this page. See
[API keys](/docs/mcp/api-keys).
# Local memory (https://docs.ctxpipe.ai/docs/mcp/local-memory)
Local memory builds on features of coding agents (Cursor, Claude Code, Codex,
OpenCode, VS Code) that change from month to month, and on ideas about agent
memory that are still settling. Expect this page and `ctxpipe memory init` to
change frequently. After you upgrade the CLI, run `npx ctxpipe memory init` again
to get the latest hooks, rules, and skills.
## What it is
Local memory is a folder of Markdown files, `.ai/memory/`, that lives in your
repository. Coding agents read it when they start a task and add to it when they
learn something worth keeping: a convention, a decision, or a correction from an
engineer.
It is plain files in Git. There is no database and no background service, and
nothing leaves your machine until you commit and push.
## Why it exists
Agents forget. Every new chat starts from scratch. Engineers then repeat
the same corrections ("we use pnpm here", "don't call that API directly"). Local
memory keeps those lessons:
* **Across chats and tools.** What one agent learns, the next one reads, whether
it runs in Cursor, Claude Code, Codex, OpenCode, or VS Code.
* **With your team.** Memory is committed and reviewed like code, so a lesson one
engineer teaches reaches everyone once it merges.
* **In ctxpipe.** When the repository is linked to a Workspace, merged lessons
become part of the knowledge of that Workspace. The MCP read tools also pass
them on to agents that work in other repositories.
## How it works
1. **The agent notices.** When you tell an agent something that sounds lasting
("from now on…", "we decided…"), a small hook saves it as a *candidate* in
`.ai/memory/events/`. Candidates stay on your machine. That folder is
gitignored.
2. **The agent decides.** When the agent finishes its turn, the hook shows it the
candidates. The agent writes the ones worth keeping into the right file (for
example `lessons-learned.md`, or a new decision record) and discards the rest.
Hooks never write lasting memory themselves.
3. **You commit it with the work.** Memory changes go in the same commit and pull
request as the code they came from. If memory stays uncommitted, the agent
gets one reminder. It gets another reminder if a later commit leaves it out.
4. **The pull request tells the story.** The agent summarizes the work in the
pull request description: what changed, why, what was tried and ruled out, and
what is left. That description is reviewed with the code.
5. **ctxpipe reads what merges.** After the merge, ctxpipe reads
`lessons-learned.md` as instructions for how your team works, and decision
records as decisions. Where pull request mirroring is on, it also reads pull
request descriptions as the history behind each change.
## What each agent gets
Paths are for repository scope. With `--scope user`, the same files go in your
home directory.
| Agent | Capture and reminders | Instructions |
| ----------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Cursor | Hooks in `.cursor/hooks.json` | `.cursor/rules/ai-memory.mdc` and skills in `.cursor/skills/` |
| Claude Code | Hooks in `.claude/settings.json` | `.claude/rules/ai-memory.md` and skills in `.claude/skills/` |
| Codex | Hooks in `.codex/config.toml` | A block in `AGENTS.md` |
| VS Code (Copilot) | Hooks in `.github/hooks/ctxpipe-memory.json` | `.github/copilot-instructions.md` and `.github/instructions/ctxpipe-memory.instructions.md` |
| OpenCode | Plugin in `.opencode/plugins/ctxpipe-memory.js` | `.opencode/memory-capture.md`, `AGENTS.md`, and an `instructions` entry in `opencode.json` |
`memory init` always writes the Cursor rule and skills in the repository, for
each choice of agents.
* **Codex** runs project hooks only after you trust the project and approve the
hooks in `/hooks`.
* **VS Code** agent hooks are in preview.
* **OpenCode** has no end-of-turn hook yet, so the plugin posts the reminder as a
new message after the agent finishes.
* **Cursor and VS Code** can also run Claude Code hooks from
`.claude/settings.json`. ctxpipe detects this so capture runs once per turn.
## Install
```bash
npx ctxpipe memory init --agents cursor,claude --scope repo --non-interactive
```
Or with the main setup wizard, which also configures the ctxpipe MCP:
```bash
npx ctxpipe init --org acme --agents cursor,claude --scope repo --memory --non-interactive
```
`--agents` takes any of `cursor`, `claude`, `codex`, `opencode`, and `vscode`.
`--scope` is `repo`, `user`, or `both`. Without prompts, the default is `repo`.
The interactive wizard asks.
## What's in the folder
```
.ai/memory/
README.md # how agents should use this folder
index.md # map of the stores below
lessons-learned.md # confirmed conventions and corrections
glossary.md # project terms
product-context.md # product and architecture overview
decisions/ # decision records (ADR-*.md) + index.md
PRDs/ # product requirements + index.md
sessions/ # summaries of work that has no pull request
events/ # candidate inbox, gitignored
```
## Local memory and the ctxpipe MCP
| | What it does |
| -------------------------- | --------------------------------------------------------------------------------------- |
| `ctxpipe` MCP (read tools) | Searches and reads your Workspaces: their knowledge files, connectors, and linked code. |
| Local memory (this page) | Markdown in one repository plus agent hooks. It is not an MCP server. |
You can use local memory on its own (`memory init`) or alongside the ctxpipe MCP
(`init --memory`). Local memory keeps the lessons of one repository. When you link that
repository to a Workspace, the MCP serves the merged lessons to every
agent in your organization.
## Finding things in memory
Agents start from `index.md` and each store's own `index.md`. For anything else
they search with ripgrep, skipping the candidate inbox:
```bash
rg -i "keyword" .ai/memory --glob '*.md' --glob '!events/**'
```
The installed `memory-search` skill describes the full procedure.
## Checking and upgrading
* `npx ctxpipe memory status` and `npx ctxpipe memory doctor` show whether memory
is set up, and list the memory files that are not committed yet.
* The hooks call `npx ctxpipe memory capture`. Agents use
`memory capture promote ` and `memory capture dismiss ` to keep or
discard a candidate. You do not run these commands yourself.
* Re-run `npx ctxpipe memory init` after upgrading the CLI. It refreshes the
rules, skills, and hooks, and keeps your own memory files. It also removes
retired pieces from older versions: the `ctxpipe-memory` MCP entry, old memory
skills, and an AgentMemory-era `README.md`. It also moves `patterns.md` into
`lessons-learned.md`.
# Connect your agent (https://docs.ctxpipe.ai/docs/mcp/mcp-docs)
ctxpipe has one remote MCP server for every organization:
```text
https://app.ctxpipe.ai/mcp
```
A self-hosted deployment uses the same `/mcp` path on its own origin. The
server uses Streamable HTTP. It gives your agent
[seven read-only tools](/docs/mcp/use-from-your-agent).
## Install with the CLI
Run this in a repository:
```bash
npx ctxpipe init
```
The wizard signs you in, asks which organization and agent clients to
configure, and shows the planned file changes before it writes them. Write the
configuration to the repository, to your user profile, or to both. To share the
setup with your team, commit the repository configuration that the CLI writes.
It holds no secret.
The wizard can also set up [local memory](/docs/mcp/local-memory). To configure
the MCP only, run `npx ctxpipe mcp add`.
| Client | Repository configuration | User configuration |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Claude Code** | `.mcp.json` | Runs `claude mcp add` when the Claude CLI is installed. Without the Claude CLI, the CLI writes the repository `.mcp.json`, or prints the command for an API key. |
| **Cursor** | `.cursor/mcp.json` | `~/.cursor/mcp.json` |
| **Codex** | No repository file. Prints a `codex mcp add` command, which writes your user configuration. For an API key, it prints the `.codex/config.toml` lines. | Runs `codex mcp add` when the Codex CLI is installed. Without the Codex CLI, the CLI prints the command. For an API key, it prints the `config.toml` lines. |
| **OpenCode** | `opencode.json` | `~/.config/opencode/opencode.json` |
| **VS Code / Copilot** | `.vscode/mcp.json` | Prints a VS Code install link |
The CLI sign-in is only for setup. The CLI keeps it in the OS keychain, or in
`~/.config/ctxpipe/` when no keychain is available. Your agent client asks you
to authorize ctxpipe separately the first time it connects.
### Scripts and CI
Use flags to run the CLI without prompts:
```bash
npx ctxpipe init --org acme --agents cursor,claude --scope repo --non-interactive
npx ctxpipe mcp add --client opencode --scope user --non-interactive
npx ctxpipe doctor --json
```
Run `npx ctxpipe init --help` for all flags. Examples are `--base-url`,
`--dry-run`, `--json`, `--no-mcp`, and `--auth api-key`. For an agent that
cannot sign in through a browser, see [API keys](/docs/mcp/api-keys).
## Authorize the client
The first time your client connects, it opens ctxpipe in the browser:
1. Sign in to ctxpipe.
2. Select the organizations that the client can read. The organization of your
current session is selected. Select at least one.
3. Leave **Also allow changes** clear. It shows only when a client asks for
write access, and the MCP tools never need it.
4. Approve. The client returns to your agent, ready to use.
Each tool call names one of the organizations that you selected. To change the
set later, or to revoke the client, open
[Connected apps](/docs/mcp/connected-apps).
## Configure a client by hand
Use manual configuration when the CLI does not support your client, or when
another tool manages the configuration. The client opens the browser to
authorize, so the configuration holds no token.
### Claude Code
`.mcp.json` in the repository:
```json
{
"mcpServers": {
"ctxpipe": {
"type": "http",
"url": "https://app.ctxpipe.ai/mcp"
}
}
}
```
For your user profile, run
`claude mcp add --transport http ctxpipe --scope user https://app.ctxpipe.ai/mcp`.
### Cursor
`.cursor/mcp.json` in the repository, or `~/.cursor/mcp.json`:
```json
{
"mcpServers": {
"ctxpipe": {
"type": "http",
"url": "https://app.ctxpipe.ai/mcp"
}
}
}
```
### Codex
```bash
codex mcp add ctxpipe --url "https://app.ctxpipe.ai/mcp"
```
### OpenCode
`opencode.json` in the repository, or `~/.config/opencode/opencode.json`:
```json
{
"mcp": {
"ctxpipe": {
"type": "remote",
"url": "https://app.ctxpipe.ai/mcp",
"enabled": true
}
}
}
```
### VS Code / Copilot
`.vscode/mcp.json` in the repository:
```json
{
"servers": {
"ctxpipe": {
"type": "http",
"url": "https://app.ctxpipe.ai/mcp"
}
}
}
```
For your user profile, use the install link that
`npx ctxpipe mcp add --client vscode --scope user` prints.
### Other clients
Use the endpoint URL with the Streamable HTTP transport. The name of the
transport key changes between clients. Read the documentation of your client.
## Check the connection
Ask your agent:
```text
Call the ctxpipe list_workspaces tool, then search_knowledge for our
standards on authentication.
```
A finished browser sign-in does not prove tool access. The tools must return
your Workspaces.
## Troubleshooting
* **The client asks for authorization after the CLI signed you in.** This is
correct. The CLI sign-in and the client authorization are separate.
* **The connection fails with only a URL.** Add the `type` field for your
client. Most clients do not detect the transport.
* **Cursor or Claude Code refuses the server type.** Use `"type": "http"`.
* **OpenCode shows the server, but it does not connect.** Use the top-level
`mcp.ctxpipe` shape with `type: "remote"` and `enabled: true`.
* **The client cannot finish the browser sign-in.** Use an
[API key](/docs/mcp/api-keys).
* **A tool returns `org_not_found`.** See
[Connected apps](/docs/mcp/connected-apps#when-a-tool-says-org_not_found).
Old configurations can have `?orgSlug=` in the URL. The server ignores it, so
you can keep it or remove it. The current CLI writes the URL without it.
## Related
* [Use ctxpipe from your agent](/docs/mcp/use-from-your-agent): the tools and how to use them
* [API keys](/docs/mcp/api-keys): CI, review bots, and headless agents
* [Connected apps](/docs/mcp/connected-apps): change organizations or revoke a client
* [Local memory](/docs/mcp/local-memory): Markdown memory that agents keep in the repository
# Use ctxpipe from your agent (https://docs.ctxpipe.ai/docs/mcp/use-from-your-agent)
The ctxpipe MCP gives a coding agent seven read-only tools. The agent does the
reasoning and drives the search. ctxpipe returns facts with stable refs, and
runs no model for a tool call. To connect a client, see
[Connect your agent](/docs/mcp/mcp-docs).
## The tools
| Tool | What it does | Main arguments |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| `list_workspaces` | Lists the organizations and Workspaces that the connection can read. With one Workspace, it also gives its repositories, sources, and readiness. | `org`, `workspace` |
| `search_knowledge` | Searches the knowledge with natural language or keywords. | `query`, `workspaces`, `kinds`, `pathPrefix`, `sources`, `updatedAfter`, `validAt` |
| `search_code` | Searches the linked repositories in one call. | `query`, `mode` (`literal`, `regex`, or `structural`), `repositories`, `language`, `pathGlob`, `caseSensitive` |
| `find_symbol` | Finds the definition, references, callers, or callees of a symbol. | `symbol`, `relation`, `filePath`, `language` |
| `read_file` | Reads a knowledge file or a repository file by its ref. | `ref`, `startLine`, `endLine` |
| `list_files` | Lists the folders and files of a Workspace repository or a linked repository. | `repository`, `path`, `depth`, `glob` |
| `get_related` | Finds the knowledge near one or more seeds in the graph that is relevant to a question. The results are ranked and explained. | `seeds`, `query`, `intent`, `excludePredicates`, `excludeKinds` |
Each tool has the read-only hint. The tools never change your data. The
`structural` mode of `search_code` needs `language`. For `get_related`,
`intent` is one of `why`, `ownership`, `change`, `dependencies`, `structure`,
`provenance`, or `general`.
The server also gives each file as a resource, `ctxpipe://{org}/{workspace}/{ref}`.
A client that supports resources lets you @-mention a file in your prompt.
## Organizations and Workspaces
* Each call names one organization in the `org` argument. Use the slug from
your ctxpipe URL. Only `list_workspaces` can leave it out, and then it lists
each organization of the connection.
* The tools that read one Workspace take its `workspace` slug.
`search_knowledge` takes a `workspaces` list. Without a list, it searches all
Workspaces of the organization that are ready, up to 8. If the organization
has more than 8, name the Workspaces.
* When a session starts, the server sends instructions that list your
organizations and their Workspaces. In most cases, the agent can call a tool
at once.
* An organization outside the connection gives `org_not_found`. See
[Connected apps](/docs/mcp/connected-apps#when-a-tool-says-org_not_found).
## Refs
A ref is a short, stable name for a result. Use a ref from one tool in another.
* A knowledge file ref is its path in the Workspace repository, for example
`knowledge/services/payments-api.md`.
* A repository file ref is `/:`, for example
`example/payments-api:src/limits.ts`.
Each result shows the branch and the commit that ctxpipe served. ctxpipe serves
the default branch of each repository, or the pinned branch or tag.
## Exclude the repository that you work in
Your agent works on a local branch with newer files than ctxpipe has. The agent
must read that repository from disk, not from ctxpipe. `search_code` and
`find_symbol` therefore need `excludeRepositories`: put the URL or the handle
of your current repository in it, or send `[]` to exclude nothing. `read_file`
and `list_files` take the same list, as an option. A ref into an excluded
repository gives `excluded_repository`.
## Good habits for an agent
* Search the knowledge before a design choice. Ask for standards, decisions,
and owners.
* Send calls that do not depend on each other in parallel.
* Use `search_code`, then `read_file` on the refs that it returns.
* Use `get_related` with a ref that you found, and a `query` or an `intent`, to
see why a decision was made or who owns a service.
* Cite the refs in your answer, and say what still needs a check in the code.
```text
Use ctxpipe before you plan this change. Call list_workspaces, then
search_knowledge for the standards on authentication, and search_code for
the session handling code. Exclude the repository that we work in.
```
## Limits and errors
* Each response has a size limit. When a result does not fit, the tool returns
one page and a `cursor`. Send the cursor to get the next page. A cursor stops
working when the Workspace changes, and the tool returns `stale_cursor`. Then
search again.
* An error has `isError: true` and `{ code, message, retryable }`.
* Each tool needs the `ctxpipe:read` scope. API keys and the normal OAuth grant
have it. A call without it gets `403 insufficient_scope`. The MCP tools never
need `ctxpipe:write`.
## Related
* [Connect your agent](/docs/mcp/mcp-docs): install and authorize a client
* [Connected apps](/docs/mcp/connected-apps): change organizations or revoke a client
* [Workspaces](/docs/workspaces): where the knowledge is
# Data and privacy (https://docs.ctxpipe.ai/docs/resources/data-processing)
This page is about **hosted ctxpipe**. On a self-hosted deployment, you choose
the providers and the retention. The model calls go from your deployment to the
provider that you configure. The ctxpipe team never sees your prompts or your
model traffic.
## What ctxpipe reads
* The git repositories that you link to a Workspace, and the Workspace
repository itself.
* The content of the sources that you add. This is:
* Linear, Notion, Confluence, PagerDuty, Buildkite, and Plain mirrors.
* The issues and closed pull requests of linked GitHub repositories.
* The Slack threads that a person captures.
ctxpipe writes this content into the Workspace repository, which you control.
Then it indexes the content.
ctxpipe reads nothing that you did not connect.
## What you control
* You create each Workspace, link each repository, and approve each source scope
in a pull request. ctxpipe ingests nothing without one of these actions.
* ctxpipe writes only to Workspace repositories. It never writes to your code
repositories. See
[What ctxpipe writes](/docs/workspaces#what-ctxpipe-writes).
* You can unlink a repository, remove a source, or delete a Workspace at any
time. This stops new indexing. The indexed data can stay until the deletion
process of your organization is complete.
[Get in touch](/docs/getting-started/get-in-touch) for the details.
* The data of your organization is separate from the data of other
organizations.
* If you stop using ctxpipe, your Workspace repositories stay with you, with
their full history.
## Sub-processors
| Provider | Purpose |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Amazon Web Services (AWS), us-west-2 | Compute and hosting, the Postgres database, storage, logs, and traces. Traces, which may hold prompt text, are kept 30 days. Isolated sandboxes (microVMs) for Workspace chat and for ingestion. A sandbox holds a copy of the Workspace repository, or of the linked repository that it ingests. |
| Cloudflare | The edge network: DNS, TLS, the proxy that each request to ctxpipe passes through, the tunnels to AWS, and Access for our internal tools |
| OpenRouter | Model routing, only to zero data retention endpoints |
| Anthropic | AI assistants used by the ctxpipe team for engineering operations; may process production logs and traces, outside OpenRouter, so zero data retention does not apply |
| OpenAI | AI assistants used by the ctxpipe team for engineering operations; may process production logs and traces, outside OpenRouter, so zero data retention does not apply |
| Railway | Being retired. Compute and hosting, and our own observability (logs, traces, and model traces), until 2026-11-15 |
| Neon | Being retired. Postgres database until 2026-11-15 |
| Vercel | Being retired for sandboxes. Isolated sandboxes for Workspace chat and for ingestion until 2026-11-15 |
## How ctxpipe uses models
* **Workspace chat.** Your messages, and the context that the agent retrieves
(search results, file excerpts, and graph context), go to the model. See
[Where the agent runs](/docs/workspaces/chat#where-the-agent-runs).
* **Ingestion.** ctxpipe uses models to extract knowledge from parts of the
linked repositories, in a sandbox with a closed network. See
[Where extraction runs](/docs/workspaces/indexing#where-extraction-runs).
* **MCP tools.** The tools call no model. They search and read your data. Your
coding agent uses its own model.
ctxpipe does not send a whole repository to a model in one request. Retrieval
and tools select the context.
On hosted ctxpipe, each model call goes through OpenRouter to
[zero data retention (ZDR)](https://openrouter.ai/docs/guides/features/zdr)
endpoints only. Our OpenRouter account enforces this, so the model providers do
not store your prompts or train on them. See the
[OpenRouter privacy policy](https://openrouter.ai/privacy).
## Retention
We do not publish fixed retention periods for indexed data yet. One exception:
ctxpipe deletes the sandbox files of a Workspace chat conversation when its
sandbox stops, 5 minutes after the last use of the conversation. The branch of
the conversation in your repository stays. For deletion times, export, or a DPA,
[get in touch](/docs/getting-started/get-in-touch).
## What we do not do
* We do not train models on your content.
* We do not share data across organizations, and we do not sell it. The
sub-processors above only operate the product.
* We do not ingest sources that you did not connect.
## Certifications
SOC 2 and ISO 27001 for ctxpipe are on the roadmap. Our infrastructure
sub-processors publish their own compliance and security documents.
For questions about your data, sub-processors, or contracts,
[get in touch](/docs/getting-started/get-in-touch).
# FAQ (https://docs.ctxpipe.ai/docs/resources/faq)
## Product
### What is ctxpipe?
Context management for software factories: the coding agents, and the
harnesses that run them, that a team uses to write software. ctxpipe keeps
your code, decisions, tickets, docs, and incidents as Markdown in a git
repository that you own. It keeps the repository current with background jobs.
It serves the repository to each coding agent through one MCP endpoint. See
[Getting started](/docs/getting-started).
### How is ctxpipe different from giving an agent more MCP servers?
More servers give an agent more to read, not a way to tell what is true.
ctxpipe gives each agent the same reviewed context: files in git with sources,
history, and links to the code. Your agent cites the files that it used. See
[Use ctxpipe from your agent](/docs/mcp/use-from-your-agent).
### Does ctxpipe train on my code?
No. On hosted ctxpipe, each model call goes only to zero data retention
endpoints. See [Data and privacy](/docs/resources/data-processing).
## Connections
### Which sources can I connect?
GitHub, Linear, Notion, Confluence, PagerDuty, Buildkite, Plain, and Slack, plus
public repositories on any git host. Linear, Notion, Confluence, PagerDuty,
Buildkite, and Plain mirror the scope that you approve. Slack captures one
thread when someone mentions the bot. See [Connections](/docs/connections).
### Is ctxpipe only for GitHub?
ctxpipe writes Workspace repositories through the GitHub App, and reads private
code through it. Other git hosts work as read-only sources: ctxpipe reads a
public repository on any host with no credential. A private repository on
another host does not work yet. See
[Plain git repositories](/docs/workspaces/linked-repositories#plain-git-repositories).
### What is the difference between a connection and a source?
A connection is the authorization for one tool. A source is a file in a
Workspace that names the connection and the scope. One connection can feed many
Workspaces.
### Does ctxpipe scan my laptop?
No. ctxpipe reads only the repositories and tools that you connect. It installs
nothing on developer machines. [Local memory](/docs/mcp/local-memory) is
Markdown in your repository, and it leaves your machine only when you push it.
### Does ctxpipe write to my code repositories?
No. ctxpipe only reads linked code repositories, and writes only to Workspace
repositories. See
[Code repositories stay read-only](/docs/workspaces#code-repositories-stay-read-only).
## MCP
### How does a tool call choose the organization?
The `/mcp` URL is the same for each organization. Each tool call names one
organization in the `org` argument. ctxpipe allows the call only when the
organization is in the set that you selected for the client, and you are still
a member. The tools have no organization id that an agent can change. See
[Use ctxpipe from your agent](/docs/mcp/use-from-your-agent).
### Which transport does the MCP use?
Streamable HTTP. A client that supports only the old SSE transport can fail.
### Can an agent change data through the MCP?
No. All seven tools are read-only.
## Account and organization
### How do invitations work?
An owner or admin sends an email invitation from **Organization settings →
Members**. See [Invite by email](/docs/managing-organization#invite-by-email).
### Can one person belong to many organizations?
Yes. Switch with the organization switcher. The URL always names the
organization. An MCP client can read each organization that you select for it.
### Can I verify my company domain?
Not yet. Access to an organization comes from invitations and roles. If you need
domain verification, [get in touch](/docs/getting-started/get-in-touch).
## Self-hosting
### What is the fastest way to evaluate a self-hosted deployment?
Docker Compose: `pnpm start` runs the `deploy` profile. See
[Docker](/docs/self-hosting/deployment/docker).
### Which databases does ctxpipe need?
PostgreSQL with pgvector, and a graph database. FalkorDB is the default. Neo4j,
Memgraph, and Amazon Neptune also work. See
[Graph databases](/docs/self-hosting/graph-databases).
### Which model providers work?
Each OpenAI-compatible provider, for example OpenRouter, Azure OpenAI, or
Ollama. Amazon Bedrock also works. See [Models](/docs/self-hosting/models).
# Open source (https://docs.ctxpipe.ai/docs/resources/open-source)
ctxpipe is open source under the **MIT License**. The core includes each job
that writes to your Workspace, so you can read exactly what changes your
knowledge. You can use, copy, change, and distribute the software. The
`LICENSE` file in the repository is the binding text. This page is not legal
advice.
## Repository
The source is at
[github.com/ctxpipe-ai/ctxpipe](https://github.com/ctxpipe-ai/ctxpipe), a pnpm
and Turborepo monorepo:
| Path | What it is |
| -------------------------- | ----------------------------------------------------------------------------------------- |
| `apps/backend` | The API, the MCP server, auth, the database schema, and the background jobs (Hono on Bun) |
| `apps/ui` | The web app (TanStack Start and React) |
| `apps/codesearch` | Code search, code navigation, and repository checkout (Zoekt and SCIP) |
| `apps/forge-ctxpipe-agent` | The Atlassian Forge app for Confluence |
| `apps/docs` | This documentation site |
| `packages/cli` | The `ctxpipe` CLI for MCP and local memory setup |
| `packages/aws-cdk` | The AWS CDK construct for self-hosting |
| `plugins/ctxpipe` | The Claude plugin |
| `examples/` | Runnable examples, such as an AWS CDK deployment |
## Hosted and self-hosted
Hosted ctxpipe runs the same open-source core. Self-host it to control your
data and infrastructure. Or use hosted ctxpipe, where we operate it for you.
See [Self-hosting](/docs/self-hosting).
## Contribute
You can open issues and pull requests on GitHub. Read `CONTRIBUTING.md` for
local development. Before a large change, read the architecture decisions in
`.ai/memory/decisions/`, so that your proposal agrees with them.
## Report a security problem
Do not report a security problem in a public issue, and do not post secrets.
Use a private GitHub security advisory on the repository, or the contact
channels on [ctxpipe.ai](https://ctxpipe.ai).
# Workspace chat (https://docs.ctxpipe.ai/docs/workspaces/chat)
Workspace chat is a coding agent that works on one Workspace. It answers from
the knowledge files, the knowledge graph, and the linked code of the Workspace,
and it can edit the Workspace repository for you.
## Start a conversation
Open a Workspace and type in the composer. Your first message creates the
conversation and gives it its own URL. The sidebar shows your five most recent
conversations under each Workspace.
```text
I am new here. What are the main services, and which files should I read first?
```
```text
Which services depend on the billing events stream, and where is that implemented?
```
```text
Before we change authentication, which decisions and standards apply?
```
To stop a reply, choose **Stop**. If a reply stops before it ends, choose
**Retry** to send your question again.
## How chat finds answers
Each answer can use:
* the **knowledge files** of the Workspace, and the content that connectors
mirror;
* the **knowledge graph** built from the links and claims in those files;
* **code search** across the
[linked repositories](/docs/workspaces/linked-repositories);
* the earlier messages of the conversation.
Chat sees only this Workspace. It does not search other Workspaces of your
organization.
## Change files
Ask the agent to write or update knowledge:
```text
Write a knowledge unit for the checkout service from what you found, and link it to the payments service.
```
While the agent works:
* **Files** shows the Workspace repository as this conversation sees it. You
can edit, create, rename, and delete files there yourself.
* **Diff** shows all the changes of the conversation, or the changes of one
turn. Use **Split view** to compare the two versions side by side. Each turn
that changes files also shows a card with its changes in the thread.
Each conversation works on its own branch, `ctxpipe/chat//`,
and never pushes to the default branch. The agent commits its work when it
finishes a task.
## Publish with a pull request
The header of the conversation has these actions:
* **Sync** (the cloud-upload icon) shows when the sandbox has changes that
GitHub does not have. It commits all changes in the sandbox, yours and the
agent's, and pushes the branch of the conversation.
* **Create PR** opens a pull request from the branch of the conversation to
the default branch.
* **Show PR** opens that pull request while it is open. Later pushes go into
the same pull request.
Review and merge the pull request like any other. After the merge, the
Workspace indexes the new commit, and the next message continues on a new
branch (`…/`) from the new default branch. A pull request that is closed
without a merge keeps its branch. Choose **Create PR** again to open a new one.
If the default branch moves while a conversation is open, ctxpipe brings the
conversation up to date before the next message. Until then, a conversation
with its own changes shows **The Workspace has new commits**, and you cannot
publish it. Send a message to bring it up to date. If the update has a
conflict, the agent resolves it.
## Where the agent runs
Each conversation runs in its own isolated sandbox, with a copy of the
Workspace repository. The first message can take longer while the sandbox
starts (**Setting up sandbox**). Later messages use the same sandbox.
On hosted ctxpipe, each conversation gets its own
[Amazon Bedrock AgentCore](https://aws.amazon.com/bedrock/agentcore/) session,
in its own microVM. No other conversation or organization shares it.
* **Commits stay.** A sandbox stops after 5 minutes without use. Before it
stops, ctxpipe pushes the commits that are not on GitHub to the branch of
the conversation. Your next message starts a new sandbox from that branch.
Changes that nobody committed are deleted.
* **Closed network.** The agent can reach GitHub, the ctxpipe model proxy,
and the Workspace tools. It cannot reach another host.
* **No credential is in the sandbox.** A proxy outside the sandbox adds
each credential to a request on its way out. The GitHub token can only read.
It is short-lived, and ctxpipe revokes it when the sandbox stops. Access to
the model and the Workspace tools is added only while a reply runs.
The sandbox never holds a credential that can push to your default branch.
Model calls, Workspace tools, and pushes go through ctxpipe, limited to that
conversation.
## Sandbox limits
An organization runs at most 50 sandboxes at once. Chats, background jobs, and
the preparation of Workspace sandboxes all count. Ingestion can use at most 40,
so chat always has 10.
When your organization is at the limit, chat shows **Workspace chat is at
capacity**. Wait a few minutes and try again. An ingestion that waits shows
**Waiting for a free sandbox**, and starts by itself when a slot is free.
These limits keep the work of an organization fair. They are not spend limits.
If they are not enough for your team,
[contact us](/docs/getting-started/get-in-touch).
## Check important answers
Chat can miss a file, read an index that is a few minutes behind, or state a
relationship too strongly. For important answers:
* ask which files support the answer;
* ask what is verified and what is inferred;
* check the indexing status in **Settings**;
* read the current code before a risky change.
Each person who can read the Workspace repository can read the content that
connectors mirrored into it. The access settings of the repository therefore
also control who can read captured Slack threads and mirrored Linear, Notion,
or Confluence content.
## Use the same context from your coding agent
Coding agents read the same Workspaces through the
[ctxpipe MCP](/docs/mcp/mcp-docs). The MCP tools search and read the Workspace
directly. They do not start a Workspace chat, so no MCP call adds a
conversation to the sidebar.
# Create a Workspace (https://docs.ctxpipe.ai/docs/workspaces/create-workspace)
Add Workspace has two steps. Step 1 chooses the **Workspace repository**. When
you choose it, the Workspace exists and ctxpipe starts to read it. Step 2 adds
the code repositories and tools that the Workspace reads. Step 2 is optional.
## Before you start
* Connect GitHub on **Connectors** if you want to select or create a repository
on GitHub. ctxpipe can write only to the repositories that the ctxpipe GitHub
App can access.
* Choose the repository for the knowledge of this Workspace. An empty
repository is good. An existing documentation repository is good too.
## Create the Workspace
### Open Add Workspace
Choose **+** next to **Workspaces** in the sidebar. You can also choose
**Create Workspace** on Home, or on the GitHub setup page after you connect
GitHub.
### Choose the repository
Choose one of these options:
* **Select GitHub**: search the repositories that the ctxpipe GitHub App can
see.
* **Create on GitHub**: open the new-repository page of GitHub. Create the
repository, make sure that the App can access it, then return and
select it.
* **Paste URL**: use any git URL. ctxpipe cannot write to a repository that
is not on GitHub, so that Workspace is read-only.
To keep the Workspace in one folder of a larger repository, type the folder
in **Root folder**, for example `docs/context`. See
[Root folder](#root-folder).
Choose **Create Workspace**.
### Add code repositories and tools
ctxpipe checks that it can write to the repository and sets it up. Then it
shows the **Sources** of the new Workspace:
* Choose **Add repositories** under **GitHub** to link
[code repositories](/docs/workspaces/linked-repositories).
* Choose **Add** next to a connected tool, such as Linear or Notion.
* Choose **Add another connection** (or **Add connection**) to connect a
new tool. This closes Add Workspace.
Each change opens a pull request that adds a source file under `sources/`.
The change applies after you merge it. You can make the same changes later
in **Settings**. Choose **Open Workspace** (or **Done**) to finish.
If the repository has ctxpipe content from an earlier version, the step
shows **Upgrade this Workspace before you add sources**. See
[Upgrades](/docs/workspaces/settings#upgrades). If ctxpipe cannot write,
the step shows the reason. See
[Write access](/docs/workspaces/settings#write-access).
### Wait for the first index
ctxpipe reads the default branch and builds the index. You can start a
conversation at once. The answers improve when indexing is done.
A repository can be the Workspace repository of only one Workspace in an
organization. Other Workspaces can link it as a linked repository.
## What ctxpipe adds to the repository
On a writable repository, ctxpipe makes one setup commit that adds these files
if they are missing:
* `AGENTS.md`, with the Workspace `name` in its front matter and a short folder
map;
* `.agents/skills/ctxpipe-knowledge/SKILL.md`, which tells coding agents how to
write knowledge files for this Workspace;
* `.ctxpipe/workspace.yaml`, which holds the
[repository format](/docs/workspaces/settings#repository-format), on a
repository that has no ctxpipe content yet.
ctxpipe keeps the files that exist. It adds only what is missing.
## Root folder
The root folder is the folder of the repository that holds the Workspace. The
default is the root of the repository.
* ctxpipe reads and writes only below the root folder. This applies to the
knowledge files, `AGENTS.md`, `sources/`, `.ctxpipe/workspace.yaml`,
connector mirrors, ingestion, and conversation commits.
* The paths in **Files**, in conversations, and in the MCP tools are relative
to the root folder.
* ctxpipe refuses a conversation push that changes a file outside the root
folder.
* The root folder is a relative path, such as `docs/context`. It cannot start
with `/` or contain `..` or `.git`. It can contain only letters, numbers,
`.`, `_`, `-`, and `/`.
When you change the root folder in **Settings**, ctxpipe reads the Workspace
again from the new folder. The current index serves until the new one is
ready. Files in the old folder stay where they are.
# Knowledge graph (https://docs.ctxpipe.ai/docs/workspaces/graph)
The **Graph** tab shows the knowledge graph of the current Workspace. Each node
is a knowledge file, and each edge comes from a link or a claim in those files.
See [Knowledge files](/docs/workspaces/knowledge-files). The knowledge that
ctxpipe extracts from linked code is in knowledge files too, so it is in the
graph. The graph shows after the first index of the Workspace. While ctxpipe
builds it, the tab shows **Graph, building**.
ctxpipe builds the graph from files. To fix the graph, fix the files: correct a
claim, add a link, or set `kind`. Then push. The graph updates when the
Workspace re-indexes.
## Read the graph
* Nodes are colored by `kind` and sized by how connected they are.
* Edges are links (`LINKS_TO`) and claims (`DEPENDS_ON`, `PART_OF`, `OWNS`,
and so on).
* The activity timeline groups relationships by when they became valid. Drag
across it to filter the graph to a time range.
Use the canvas controls to zoom, reset the view, pause or play the layout,
lasso-select a group of nodes, or capture a screenshot.
## Inspect a node
Select a node to open its details:
* **Path**: the file behind the node. **Go to definition** opens it in the
**Files** tab; **Copy path** copies it.
* **Summary**, **Connections**, and **Strongest connections**.
* **Predicates** and **Neighbor kinds**: what kinds of relationships and
nodes surround it.
* **Activity**: when its relationships were valid.
Review a node with few or no connections. It is often a stub or a stale unit.
Lasso-select several nodes to see a summary of the selection: node and edge
counts, kinds, and predicates.
## Treat the graph as a guide
The graph shows what the files claim. Confidence and dates help you decide
where to look, but they are not proof. Before you act on an important
relationship, open the file and the source it points to.
# Workspaces (https://docs.ctxpipe.ai/docs/workspaces)
A **Workspace** holds one body of team knowledge, for example one product area
or one team. Each Workspace has:
* one **Workspace repository**: a git repository of Markdown knowledge files,
connector content, and `AGENTS.md`;
* zero or more **linked repositories**: code repositories that ctxpipe indexes,
so that chat and agents can search them.
An organization can have many Workspaces. Each member of the organization can
open each Workspace.
## Git is the source of truth
All that ctxpipe knows about a Workspace is in files in its Workspace
repository. ctxpipe reads those files to build its search index and knowledge
graph, and writes new knowledge back as ordinary commits. Because of this:
* you read, review, and edit the knowledge with the tools that you already use;
* each change that ctxpipe makes is a commit in the history of the repository;
* if you stop using ctxpipe, the repository and its knowledge stay with you.
### What ctxpipe writes
ctxpipe writes only to the Workspace repository, through the ctxpipe GitHub
App. Each change is a small commit to the default branch, or a pull request:
| Change | How it gets to the default branch |
| -------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| The [setup commit](/docs/workspaces/create-workspace#what-ctxpipe-adds-to-the-repository) of a new Workspace | A commit |
| Connector mirrors: Linear, Notion, Confluence, Slack, PagerDuty, Buildkite, Plain, and the issues and closed pull requests of linked GitHub repositories | A commit for each sync |
| [Knowledge extracted](/docs/workspaces/indexing#how-extraction-works) from linked repositories | One commit for each run, `Update extracted knowledge` |
| Upkeep, for example links that ctxpipe rewrites after a rename, a `valid_from` that a file does not have, or the automatic-upgrades setting | A commit |
| A change to `sources/`: add, edit, rename, or remove a source | A pull request that a person merges |
| A [format upgrade](/docs/workspaces/settings#upgrades) | A pull request on `ctxpipe/upgrade/format-` |
| [Workspace chat](/docs/workspaces/chat#publish-with-a-pull-request) | A branch for each conversation, and a pull request that you open |
ctxpipe never force-pushes the default branch. It force-pushes only refs that
it owns: the branch of a conversation, the upgrade branch, and the temporary
refs of an extraction run. If someone else pushed to the default branch first,
ctxpipe rebases its change onto theirs. If both changed the same lines,
ctxpipe merges the two changes before it pushes.
If the default branch is protected, connector mirrors and extracted knowledge
stop. See [Write access](/docs/workspaces/settings#write-access).
### Code repositories stay read-only
ctxpipe does not write to
[linked repositories](/docs/workspaces/linked-repositories). It reads them with
short-lived tokens that can only read. The GitHub mirror only reads issues and
pull requests, and writes them to the Workspace repository.
A linked repository can also be the Workspace repository of another Workspace
in your organization. ctxpipe then writes to it only for that other Workspace,
and does not extract knowledge from it.
## Use a Workspace
Open a Workspace from the sidebar. The main column is the conversation. The
pane on the right has these tabs:
| Tab | What it shows |
| ------------ | ----------------------------------------------------------------------------------------------------------------------- |
| **Files** | The Workspace repository, as the current conversation sees it. Browse, edit, create, rename, and delete files. |
| **Diff** | The changes of the current conversation. It shows only when a conversation is open. |
| **Graph** | The knowledge graph of the Workspace. While ctxpipe builds it, the tab shows **Graph, building**. |
| **Settings** | The name, repositories, sources, ingestion progress, and upgrades. See [Workspace settings](/docs/workspaces/settings). |
The sidebar lists your Workspaces with their five most recent conversations. A
pulse next to a Workspace shows that ctxpipe is ingesting content for it. Home
shows the recent activity and commits of all Workspaces. Choose **Search** in
the sidebar to find a Workspace or a page.
## How a Workspace stays current
When a new commit arrives on the default branch, from ctxpipe or from a person,
ctxpipe indexes it. See
[How indexing works](/docs/workspaces/indexing).
## Read-only Workspaces
ctxpipe writes to the Workspace repository only through the ctxpipe GitHub App.
If it cannot push, the Workspace is **read-only**. For example, the repository
is not on GitHub, or the App lost access. Search, chat, and the graph continue
to work on the latest commit, and writes start again when access returns.
**Settings** shows the reason. See
[Write access](/docs/workspaces/settings#write-access).
# How indexing works (https://docs.ctxpipe.ai/docs/workspaces/indexing)
**Indexing** turns the files and the linked code of a Workspace into data that
chat, search, the graph, and MCP can use. It runs in the background after each
change, so new content can take a few minutes to show in answers.
## Two kinds of indexing
### The Workspace repository
When the default branch of the Workspace repository changes, ctxpipe reads
the knowledge files at that commit and rebuilds the index: units, links and
claims for the graph, and embeddings for search. This step never changes
files and calls no extraction model. The previous index serves until the new
one is ready.
### Linked repositories
When a linked repository changes, ctxpipe checks out its linked branch and
builds code search and code navigation. Then it
[extracts knowledge](#how-extraction-works) from the new commit and
commits it to the Workspace repository, which starts the first kind of
indexing.
## Changes and full rebuilds
ctxpipe moves the search and graph data of each organization to a new
knowledge store, one organization at a time. On the new knowledge store, an
index run of the Workspace repository applies only the files that changed
after the last indexed commit. A file that you remove from the repository
leaves the index.
Some events start a **full rebuild**. A full rebuild reads all the knowledge
files at the newest commit again:
* The last indexed commit is not in the Workspace repository any more. For
example, a user force-pushed the default branch, and Git then deleted the
old commits (a garbage collection). Without that commit, ctxpipe cannot find
which files changed.
* The first index run of a Workspace.
* A new ctxpipe version that changes how ctxpipe reads files, a new embedding
model, or a new root folder of the Workspace.
* A change to the sources of the Workspace (`.ctxpipe/sources.yaml` or
`sources/`).
* Runs started for more than 10 different commits after the last indexed
commit, the current commit included, and none of them completed.
During a full rebuild:
* The run takes longer than a usual run, because ctxpipe reads and embeds each
file again.
* Chat, search, the graph, and MCP continue to answer from the index. ctxpipe
replaces the files in groups, so an answer during the run can include some
files from the old commit and some from the new commit.
* A file that is not at the newest commit leaves the index when the run ends.
* When the run completes, the index is the same as the files at the newest
commit. You do not have to do anything.
If a run fails, the next run tries the same files again.
## How extraction works
Extraction reads one linked repository at one commit. It writes services,
apps, libraries, APIs, dependencies, data stores, infrastructure, decisions,
and instructions as knowledge files.
### Prepare a sandbox
ctxpipe clones the linked repository into an isolated sandbox. It adds a
read-only copy of each Workspace repository that links the repository. See
[Where extraction runs](#where-extraction-runs).
### Run the extraction skills
Each step runs one skill. Scripts find the facts first, for example from
manifests, lockfiles, and code patterns. A model confirms the facts, or
adds what a script cannot see. Each step runs separately for each
Workspace that links the repository. It updates or links the files that
the Workspace already has, so that it does not add a second file for the
same thing. The sandbox can only stage file changes. It cannot write to a
repository.
### Check each file outside the sandbox
ctxpipe checks each staged file before it accepts it. It rejects the
file, and keeps the other files of the step, when:
* the path is not in a knowledge folder of the
[extracted files](/docs/workspaces/knowledge-files#extracted-files), or
it is in `sources/`, `.ctxpipe/`, `.agents/`, a connector mirror, or
`AGENTS.md`;
* the change breaks the ownership rules, or changes a human section;
* the front matter has an unknown key, or a claim has no `evidence` that
exists in the linked repository at that commit;
* a link does not resolve, or an external link goes to a repository that
the Workspace does not know;
* the text has control characters, hidden characters, raw HTML, or an HTML
comment;
* the file is larger than 64 KiB.
A step can delete at most 20% of the files that its source owns.
### Publish one commit
ctxpipe keeps the result of each step in a temporary ref,
`refs/ctxpipe/ingest/…`, in the Workspace repository. ctxpipe does not
index these refs. At the end of the run, ctxpipe commits the result to the
default branch as one commit, `Update extracted knowledge`, and deletes
the refs.
Extraction publishes only to a Workspace at
[format 3](/docs/workspaces/settings#repository-format) or later that ctxpipe
can write to. Below format 3, the knowledge that the Workspace has stays
readable.
## Guards against prompt injection and secrets
The content of a linked repository can try to give orders to the model. The
model gets the instruction to treat each file as data, never as instructions.
ctxpipe also checks the result:
* It rejects a file with a heading that addresses AI agents or models.
* It rejects a file with an instruction-like phrase outside a quote. A quote of
such a phrase must give a `Source:` path, and the text must be in that file
of the linked repository.
* It rejects all the files of a step when the text that the step writes has a
value that looks like a secret. Examples are a private key and a
connection string with a password. More examples are an `Authorization`
header, and a key such as `*_TOKEN` with a real value. A known token format
is also a secret. The scan also decodes base64 and percent-encoded text. The
rejection gives the rule and the line, never the value.
These guards limit what injected text can do. They cannot stop all of it.
Review extracted files like other knowledge files. The `AGENTS.md` that
ctxpipe writes tells coding agents to treat extracted files as data.
## Where extraction runs
Extraction runs in a different sandbox from
[Workspace chat](/docs/workspaces/chat#where-the-agent-runs). On hosted
ctxpipe, each extraction run has its own Amazon Bedrock AgentCore sessions.
The differences from chat:
* **Closed network.** The sandbox can fetch the linked repository and the
Workspace repositories of the run with git. It can call the ctxpipe model
proxy and the ctxpipe code search tools. It can do nothing else. It cannot
push, call the GitHub API, or reach another host or a private address. A public repository on another
git host is reachable only while the sandbox clones it, before a model runs.
* **No credential in the sandbox.** A ctxpipe proxy outside the sandbox adds
each credential to a request on its way out. The GitHub tokens can only
read, and ctxpipe revokes them when the run ends.
* **Limited agent.** The model runs as a user with no admin rights. The user
owns only the output folder `/ctxpipe/out` and the home folder
`/ctxpipe/home`, so the model can write only there. It uses only the skills
of ctxpipe. The `AGENTS.md` file and the skills in the linked repository do
not change it.
Extraction uses the sandbox slots of your organization. See
[Sandbox limits](/docs/workspaces/chat#sandbox-limits). For self-hosted
sandboxes, see
[Chat sandboxes](/docs/self-hosting/configuration#chat-sandboxes).
## Re-ingestion
When a new ctxpipe version changes what extraction writes, **Settings** offers
the upgrade **Fresh re-ingestion of the source repositories**. It runs the
extraction again on each linked repository. See
[Upgrades](/docs/workspaces/settings#upgrades).
## Read the status
**Settings** in each Workspace shows the hydrate status of the Workspace on the
**Knowledge graph** card, and the status of each linked repository on the
**Sources** card. A row of the **Sources** card that ingests now shows what
ctxpipe does and how long it has run. A button on the row opens the stage and
the progress of each run. In the sidebar, a pulse next to a Workspace shows that
it ingests now. A re-ingestion that you start from an upgrade shows on the
**Upgrades** card.
For the status of a linked repository (**Pending**, **Indexing**, **Indexed**,
**Indexed with issues**), see
[Indexing status](/docs/workspaces/linked-repositories#indexing-status). For the
hydrate status, see
[Hydrate and skipped files](/docs/workspaces/settings#hydrate-and-skipped-files).
When an index of the Workspace fails, the last good version continues to serve.
Fix the cause, often front matter that is not valid or lost repository access,
and push again.
Indexing builds search and graph data from files and from claims that a
model extracted. It does not prove an answer. Ask for the files, and check
risky decisions against the current code.
## Three layers
Content in a Workspace lives in three layers:
1. **Files**: the bytes in git, in the Workspace repository or a linked
repository.
2. **Search**: the text and code index over those files. Chat and MCP can
retrieve anything here.
3. **Graph**: typed nodes and relationships from links, claims, and `kind`
front matter. Mirrored connector content becomes `Issue`, `Thread`,
`PullRequest`, and similar nodes. It is never treated as instructions.
Cross-tool links are joined by URL or identifier, not guessed.
# Knowledge files (https://docs.ctxpipe.ai/docs/workspaces/knowledge-files)
The knowledge of a Workspace is plain Markdown in its Workspace repository.
ctxpipe keeps no hidden copy that can drift. Chat, search, and agents use what
is on the default branch.
## One file, one unit
Each Markdown file is one **knowledge unit**. Its path is its identity, so
renaming a file creates a new unit. Keep units short and focused, and link to
other units rather than copying them.
On an empty repository, use `knowledge//.md`. On an existing
repository, follow the folders already there.
```text
workspace-repository/
├── AGENTS.md # Workspace name + folder map
├── .agents/skills/ctxpipe-knowledge/
│ └── SKILL.md # How agents should write knowledge here
├── knowledge/
│ ├── services/checkout.md
│ └── decisions/payments-provider.md
├── .ctxpipe/workspace.yaml # Repository format (host-only)
├── sources/ # One file for each source
│ ├── github/acme/checkout.md
│ └── linear/engineering.md
├── linear/ notion/ confluence/ # Connector mirrors
└── slack/ pagerduty/ plain/ github/
```
## Relationships
The knowledge graph has two layers.
**Links.** A relative Markdown link from one file to another becomes a
`LINKS_TO` relationship:
```markdown
Checkout calls the [payments service](../services/payments.md).
```
**Claims.** Optional `claims:` front matter states typed facts about the unit:
```markdown
---
kind: Service
claims:
- to: ../services/payments.md
predicate: DEPENDS_ON
confidence: 0.7
valid_from: 2026-03-01
---
# Checkout
```
| Field | Meaning |
| ------------------------ | -------------------------------------------------------------------------------------------------- |
| `to` | Required. File-relative path of the related unit. Items without `to` are skipped. |
| `predicate` | The relationship, for example `DEPENDS_ON`, `PART_OF`, `OWNS`, `SUPERSEDES`. |
| `confidence` | How sure the author is: about `0.5` typical, `0.7` strong, `0.85` and above rare. |
| `valid_from`, `valid_to` | When the fact holds. A missing `valid_from` defaults to the time of the commit that introduced it. |
| `source` | Optional pointer to where the fact came from. |
`kind` is optional and sets the unit's node type in the graph (`Service`,
`API`, `Library`, `Decision`, `Pattern`, and so on). Without it, the unit
appears as a generic knowledge unit.
Older facts count for less over time. Recent, well-sourced facts rank highest
in answers.
## Skipped files
ctxpipe skips a file that has one of the errors below, and continues with the
other files. One bad file does not stop the Workspace. A skipped file stays in
**Files**, because it is in git, but chat, MCP, search, and the graph do not
use it. The **Knowledge graph** card in **Settings** lists the skipped files of
the latest commit under **ctxpipe skipped N files**.
| Reason | Cause | Fix |
| ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| Front matter is not valid | The front matter has no closing `---` line, or its YAML is not valid. | Correct the front matter. |
| Repeats a linked repository or the Workspace's own repository | Shows only in results from an earlier version, for a `repositories/*.md` file. ctxpipe no longer reads these files. | None. |
Push the fix to the default branch. When ctxpipe reads the new commit, the file
is a knowledge unit again, and it leaves the list.
Other errors stop the read of the whole commit. For example, a set of files
that is too large shows **The knowledge graph did not load** in **Settings**.
See
[Workspace settings](/docs/workspaces/settings#hydrate-and-skipped-files).
## Sources and linked repositories
Each file under `sources/` is a source. The GitHub source files link the code
repositories of a Workspace. See
[Linked repositories](/docs/workspaces/linked-repositories) and
[Workspace settings](/docs/workspaces/settings#sources). ctxpipe does not read
files in `repositories/`. When an organization moves to Workspaces, the upgrade
pull request turns those files into source files.
## Extracted files
ctxpipe writes the knowledge that it
[extracts from linked repositories](/docs/workspaces/indexing#how-extraction-works)
only to these folders: `knowledge/codebases/`, `services/`, `apps/`,
`libraries/`, `apis/`, `operations/`, `databases/`, `infrastructure/`,
`streams/`, `decisions/`, `teams/`, and `instructions/`. ctxpipe sets these
front matter fields. Do not change them by hand:
| Field | Meaning |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `generated_by` | `ctxpipe-extract`. ctxpipe extracted the file. |
| `sources` | The source files that own the file, for example `sources/github/acme/checkout.md`. |
| `source_sha` | The commit of the linked repository that ctxpipe read. |
| `root` | The folder of the package in the linked repository. It is `.` for the whole repository. Only `Repository`, `Service`, `App`, and `Library` files have it. |
| `skill`, `bundle` | The extraction step that wrote the file, and the version of the extraction. |
Each claim of an extracted file has `evidence`: a path in the linked
repository, and `script` or `model`. ctxpipe sets `confidence` from it: `0.7`
for `script` and `0.5` for `model`.
To change an extracted file:
* Put your own text between `` and
``. Ingestion never changes it.
* To take over the file, remove `generated_by`. After that, ingestion can only
add claims and new lines in its `## Related` section.
A file that a source owns alone can change fully on the next run. When you
remove a repository source, ctxpipe deletes the files that only that source
owned. ctxpipe also removes the source from the files that it shares with other
sources. A file with a human section stays. ctxpipe removes only its own fields
and claims.
For all the commits that ctxpipe makes, see
[What ctxpipe writes](/docs/workspaces#what-ctxpipe-writes).
# Linked repositories (https://docs.ctxpipe.ai/docs/workspaces/linked-repositories)
**Linked repositories** are the code repositories that a Workspace searches.
ctxpipe indexes each one for code search, and extracts its services, APIs,
libraries, and their relationships into the Workspace repository as knowledge
files.
A repository can be linked to many Workspaces. Each Workspace keeps its own
index, so a search in one Workspace never returns the code of another.
## Link repositories
Each linked repository is one source file in the Workspace repository, for
example `sources/github//.md`. A pull request adds or removes the
file. The change applies after you merge the pull request.
### Open Add repositories
In the Workspace, open the **Settings** tab. Under **Sources**, choose
**Add repositories** in the **GitHub** group.
### Choose the repositories
* **Select GitHub**: search the repositories that the ctxpipe GitHub App can
see, and select one or more. Only an organization owner or admin can list
them.
* **Paste URL**: type the URL of a public repository on any git host, or of a
public GitHub repository with no GitHub connection. See
[Plain git repositories](#plain-git-repositories).
Choose **Open a pull request**. One pull request can link up to 20
repositories.
### Merge, then wait for indexing
Merge the pull request. Until then, the row shows **Awaiting merge**. After
the merge, ctxpipe queues each repository for indexing.
To remove a repository, choose the unlink button on its row. ctxpipe opens a
pull request that deletes the source file. After the merge, ctxpipe stops
searching the repository and removes the knowledge that it extracted from it.
## Pin a branch or tag
A source reads the default branch of the repository. To read another branch or
a tag, add `ref` to the front matter with a pull request:
```markdown
---
provider: github
connection: con_example
repository: acme/payments
ref: release
---
The payments API. Use it for questions about refunds and card tokens.
```
The body is an optional description of what the repository is for. People own
the body, and ctxpipe never changes it. You can rename or move the file inside
`sources/github/`. The pin stays with the repository.
## Plain git repositories
ctxpipe can read a repository that needs no credential: a public repository on
another git host, or a public GitHub repository. Use **Paste URL**, or add a
source file under `sources/git/` with a pull request:
```markdown
---
provider: git
url: https://git.example.org/team/tools.git
---
```
ctxpipe checks the URL before it opens the pull request. On hosted ctxpipe, the
URL must:
* use `https://` and the default port;
* have a public domain name as its host, not an IP address or a name such as
`localhost`, and resolve only to public addresses;
* have no user name, password, query, or fragment.
ctxpipe does not support ssh (`git@…`, `ssh://`), `git://`, or `file://` URLs,
and does not follow a redirect. Use the exact clone URL. A repository that
needs a credential works only on GitHub, through the GitHub App.
A self-hosted deployment can also read a git server inside your network. See
[Plain git repositories](/docs/self-hosting/configuration#plain-git-repositories)
in the self-hosting configuration.
## Indexing status
| Status | Meaning |
| ----------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Pending** | Linked, waiting for indexing. |
| **Indexing** | ctxpipe is building search and extracting knowledge. Answers can be incomplete. |
| **Indexed** | Search and extraction are done for the current commit. |
| **Indexed with issues** | Search works, but part of the index failed, usually code navigation for one language. The row shows the reason. |
Indexing runs again when the linked branch or tag changes. The row of the
repository in the **Sources** card of **Settings** shows the progress of each
run.
## What indexing builds
* **Code search**: text, file, and symbol search over the linked branch.
* **Code navigation**: definitions and references for the supported languages.
* **Extracted knowledge**: services, APIs, libraries, infrastructure, and
instructions such as `AGENTS.md`, skills, and ADRs, as knowledge files in
the Workspace repository. See
[How extraction works](/docs/workspaces/indexing#how-extraction-works).
ctxpipe only reads a linked repository. It never writes to it. See
[Code repositories stay read-only](/docs/workspaces#code-repositories-stay-read-only).
## Issues and pull requests
For a linked GitHub repository, ctxpipe also mirrors its issues and its closed
pull requests into `github/issues/` and `github/pulls/` in the Workspace
repository. When you unlink, new items stop, and the files already mirrored
stay. See [GitHub connector](/docs/connections/source-connectors/github).
# Workspace settings (https://docs.ctxpipe.ai/docs/workspaces/settings)
Open a Workspace and choose the **Settings** tab. The pane has a summary line at
the top, and then one card for each part of the Workspace.
The summary line names the cards that need you, and the cards that work now.
Each name is a link to its card. The line has one of these forms:
* **Everything is up to date.**
* **N parts need you:** (or **1 part needs you:**) and the names of the cards. The line can also say
**Running:** and the names of the cards that work now.
* **Nothing needs you. Running:** and the names of the cards that work now.
A card that needs you has an amber edge. These are the cards, in the order of
the pane:
| Card | What it holds |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Upgrades** | The upgrades that wait or run, the automatic-upgrades switch, the repository format, and the history. See [Upgrades](#upgrades). |
| **Sources** | The code repositories and tools that the Workspace reads, and the live work of each one. See [Sources](#sources). |
| **Knowledge graph** | Whether ctxpipe loaded the graph, and the files that ctxpipe skipped. See [Hydrate and skipped files](#hydrate-and-skipped-files). |
| **Repository** | The Workspace repository, which this page calls the **Context repository**, the write access, and the **Root folder**. See [Repositories](#repositories) and [Write access](#write-access). |
| **Workspace** | The **Name** (stored in `AGENTS.md`) and the **URL name**. If you change the URL name, old links stop working. Choose **Save** to keep a change. |
| **Danger zone** | **Delete Workspace**. This removes the Workspace and its conversations from ctxpipe. Your repository and its files stay. |
Each member can read the pane. Only an organization owner or admin can start an
upgrade or change a source.
## Upgrades
On the **Upgrades** card, each upgrade that needs you or runs now is a row. The
dot of the row shows its kind: amber for an item that needs you, teal for work
that runs, and red for a failure. The card header shows **Up to date** when no
row is open and the format is the latest. It shows **Checking the format** while
ctxpipe reads the format file.
A re-ingestion row also shows its spend so far. You can leave the page. The work
continues. Applied upgrades show under **History**. Home shows an alert for each
Workspace that has an upgrade to finish. **Review upgrade** on the alert opens
this card.
An owner or admin chooses **Start** on a row. A member sees **Ask an admin** in
place of the button.
To stop a re-ingestion that is queued or runs, an owner or admin chooses
**Cancel** on its row, then confirms. ctxpipe stops the run and each ingestion
run of it. The run makes no more model calls and publishes nothing more. A
repository that finished before the cancel keeps its new knowledge. The row
shows **Canceled**, and ctxpipe does not start it again by itself. A format
upgrade has no **Cancel**: it only opens a pull request.
Upgrades move the Workspace repository to the layout that this version of
ctxpipe reads. Nothing changes until you merge the pull request. There are two
kinds of upgrade:
* **Upgrade to format N** changes the layout of the repository. ctxpipe opens
one pull request on the branch `ctxpipe/upgrade/format-`, with one commit
for each step. The last commit writes `.ctxpipe/workspace.yaml`. Read the
notes under the row before you merge.
* **Fresh re-ingestion of the source repositories** reads each linked
repository again and publishes the new knowledge. See
[How extraction works](/docs/workspaces/indexing#how-extraction-works). It
needs format 3. For the cost and the restart rules, see
[Upgrade a Workspace](/docs/self-hosting/upgrades#upgrade-a-workspace).
| Status | What to do |
| ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Ready to start** | An owner or admin chooses **Start**. |
| **Queued**, **Preparing the pull request**, **Waiting** | Wait. |
| **Waiting for merge** | Review and merge the pull request. If the card does not change after a few minutes, choose **Check again**. |
| **Applied** | Nothing. The upgrade is done. |
| **Failed** | Read the reason, fix the cause, then choose **Start again**. |
| **Canceled** | An owner or admin stopped the re-ingestion. To run it, choose **Start again**. |
| **Blocked** | Read the reason and remove the cause. See the table below. |
If someone closes the pull request without a merge, the upgrade goes back to
**Ready to start**.
| Blocked title | Cause |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ctxpipe cannot open the upgrade pull request | ctxpipe has no write access. See [Write access](#write-access). |
| ctxpipe cannot publish the re-ingestion | ctxpipe has no write access. See [Write access](#write-access). |
| Apply the format upgrade first | The repository is below format 3. Merge the format upgrade, then choose **Check again**. The item then shows **Waiting** until ctxpipe reads the new format. |
| This deployment cannot run the re-ingestion | The deployment has no ingest sandbox. The operator must configure one. |
### Automatic upgrades
By default, nothing starts by itself. To change this, an owner or admin turns
the **Automatic upgrades** switch on or off, or chooses **Always update** on the
Home alert. ctxpipe then commits the setting into `.ctxpipe/workspace.yaml` on
the default branch.
When automatic upgrades are on, ctxpipe starts each upgrade when it finds it. A
merge always needs a person.
### Repository format
Each Workspace repository holds `.ctxpipe/workspace.yaml` below its root
folder. The key `format` is the version of the repository layout. The optional
key `upgrades` is `manual` (the default) or `automatic`.
```yaml
format: 3
upgrades: manual
```
The format line of the **Upgrades** card compares this format with the format
that ctxpipe reads.
| Format | Layout |
| ------ | --------------------------------------------------------------------------------------------------------------------- |
| 0 | No format file: a repository from before Workspaces. |
| 1 | The knowledge that a fresh ingestion cannot rebuild is exported. Sources are in the older `.ctxpipe/sources.yaml`. |
| 2 | One Markdown file for each source under `sources/`. Source changes need format 2. |
| 3 | Ingestion writes ownership front matter in each knowledge file. This is the current format. Ingestion needs format 3. |
Only the setup commit of a new Workspace, an upgrade pull request, and the
automatic-upgrades setting write this file. A format that this version of
ctxpipe does not know, or a file that does not parse, makes the Workspace
**Read-only**. Reads continue.
## Sources
A source tells the Workspace what to read through a connection. Each source is
one Markdown file at `sources//.md` in the Workspace
repository. The front matter holds the provider, the connection, and the scope.
The body is an optional description that people and agents read. The path of
the file is the identity of the source.
The file on the default branch is the live scope. Each change to a source is a
pull request that a person merges. No agent can change `sources/` without that
review. The buttons show only when ctxpipe can open a pull request. See
[Write access](#write-access).
The **Sources** card has one group for each provider that has a connection or a
source. Each row shows the source, its file, and its status. **Add another
connection** opens **Connectors**.
While ctxpipe works on a source, its row shows what ctxpipe does, for example
**Syncing** or **Publishing**, and how long it has run. A row that has more
than one run has a button that opens the stage and the progress of each run.
You can leave the page. The work continues. While ingestion runs, the sidebar
shows a pulse next to the Workspace.
### Add, edit, rename, or remove a source
| Action | How |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Add | Choose the add button in the group of the provider (**Add repositories** for GitHub). Select the scope, then choose **Open a pull request**. |
| Edit | Open the actions menu of the row, then choose **Edit**. Change the scope, then choose **Open a pull request**. |
| Rename | Open the actions menu of the row, then choose **Rename**. ctxpipe moves the file, and updates the knowledge files that name the old path, in the same pull request. |
| Remove | Open the actions menu of the row, then choose **Remove**. After the merge, the next sync deletes the files that the source wrote. Git history keeps them. Captured Slack threads stay. |
In the add dialog, **File name** is optional. Without a name, ctxpipe names the
file from the provider, for example `linear` or `linear-2`. A GitHub source is
always `sources/github//.md`.
Only an organization owner or admin can list the content of most providers. A
member sees **Ask an admin to choose what this source syncs**. Any member can
list Confluence spaces.
A row with an open pull request shows **Awaiting merge**, **Rename awaiting
merge**, or **Removal awaiting merge**, with a link to the pull request. After
the merge, the row updates in a few seconds.
### Source status
| Status | Meaning |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Connected** | The file, the connection, and the scope work. The source syncs. The Workspace creation form shows **OK** for this status. |
| **Not valid** | The file does not parse, has a wrong shape, or breaks a rule. The sync skips it. The reason names the problem. |
| **Connection missing** | The connection of the source was removed. The file stays. |
| **Out of access** | The connection cannot reach the scope, for example a repository that is not in the GitHub App installation. ctxpipe checks again later. |
These rules apply across files:
* Two sources cannot name the same GitHub repository, or write to the same
folder. The second file is **Not valid**.
* A GitHub source cannot name the Workspace repository.
* A source file larger than 64 KiB, or a file that is not text, is **Not
valid**.
### Source file format
Each file has the front matter key `provider`. Each file except a plain git
source also has `connection`. An unknown key makes the file **Not valid**.
| Provider | Path | Scope keys |
| ---------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| GitHub | `sources/github//.md` | `repository` (`owner/name`), optional `ref` (a branch or a tag) |
| Linear | `sources/linear/.md` | `teams` (team keys), `projects` (project ids), optional `customerRequests` (`exclude` or `limited`). At least one team or project. |
| Notion | `sources/notion/.md` | `pages` and `databases`, each a list of `id` and `title`. At least one of them. |
| Confluence | `sources/confluence/.md` | `spaces`: a list of `key` and `pages` (`all` or a list of page ids) |
| Slack | `sources/slack/.md` | `channels`: a list of channel ids |
| PagerDuty | `sources/pagerduty/.md` | `services`: a list of service ids |
| Buildkite | `sources/buildkite/.md` | `pipelines`: a list of pipeline slugs |
| Plain | `sources/plain/.md` | `helpCenters` (Help Center ids) and `labels` (label type ids). At least one of them. |
| Plain git | `sources/git//.md` | `url` only |
Each path segment after the provider folder matches
`[a-z0-9][a-z0-9._-]{0,99}`. The keys `id` and `kind` are not allowed.
## Hydrate and skipped files
Hydrate is the step that reads the files of the repository at one commit and
builds the search index and the graph. The **Knowledge graph** card shows its
status:
| The card shows | Meaning |
| ------------------------------------------ | ----------------------------------------------------------------------- |
| **The knowledge graph is loaded.** | ctxpipe read the latest commit. |
| **The knowledge graph waits to load.** | ctxpipe did not start yet. |
| **ctxpipe loads the knowledge graph now.** | ctxpipe reads a commit now. The previous index serves until it is done. |
| **The knowledge graph did not load** | ctxpipe could not read the repository. The card shows the error. |
While ctxpipe indexes a new commit, the card also shows **Indexing the newest
commit…** and the time since it started.
Before the first hydrate ends, the Workspace page shows **Preparing** and the
name of the Workspace. If the first read fails, it shows **Could not prepare
the Workspace** with the reason and **Try again**.
When ctxpipe skips files, the card shows **ctxpipe skipped N files**, with
the path and the reason for each file. Chat, MCP, search, and the graph do not
use a skipped file. Select a path to open the file in **Files**. Fix the file
and push the fix to the default branch. See
[Skipped files](/docs/workspaces/knowledge-files#skipped-files).
## Repositories
* **Context repository**: choose **Change** to move the Workspace to a
different repository. The dialog has the same options as
[Create a Workspace](/docs/workspaces/create-workspace). The Workspace and its
conversations stay. The files in the old repository stay where they are.
Until the new repository is loaded, answers come from the old one.
* **Root folder**: the folder of the repository that holds the Workspace. See
[Root folder](/docs/workspaces/create-workspace#root-folder).
## Write access
ctxpipe writes to the Workspace repository through the ctxpipe GitHub App. The
**Repository** card shows the result of the last check in one line. The header
of a conversation also shows a tag, but not when ctxpipe can push. The tag has
the same title as the line.
| The UI shows | Meaning |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ctxpipe can push to this repository. | The GitHub App can push to the repository. The state is writable. |
| **Changes by pull request** | The default branch is protected. Conversations and source changes use pull requests. Connector mirrors and extracted knowledge cannot push to the default branch, so they stop. |
| **Checking write access** | ctxpipe is checking. The pane checks again by itself. On the card, the line says "ctxpipe checks write access now." If the check takes longer than usual, the card shows **Still checking write access**. |
| **Could not check write access** | GitHub returned an error or a rate limit. Choose **Check again**. |
| **Read-only** | ctxpipe cannot write. Search, chat, and the graph continue to work on the latest commit. |
A **Read-only** Workspace shows the reason:
| Reason | What to do |
| ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| The remote is not GitHub. | Select a GitHub repository as the context repository. |
| GitHub is not connected for this organization. | An owner or admin connects GitHub on **Connectors**. Then choose **Select the repository again**. |
| The repository is not in the GitHub App installation. | Add the repository to the installation in GitHub. |
| The GitHub App cannot write to this repository. | Give the GitHub App **Contents: Read and write**. |
| The Workspace needs a newer ctxpipe version. | The repository format is newer than this ctxpipe version. Upgrade ctxpipe. See [Repository format](#repository-format). |
| The file `.ctxpipe/workspace.yaml` is not valid. | Repair the file in the repository. |
After you fix the cause, reload the page. ctxpipe checks again at most once a
minute.
# Authentication (https://docs.ctxpipe.ai/docs/self-hosting/authentication)
ctxpipe uses [Better Auth](https://www.better-auth.com) for identity management.
All browser sign-in, provider callbacks, sessions, and MCP OAuth go through the
same public backend origin.
## Required auth boundary
| Variable | Requirement |
| ---------------------- | ------------------------------------------------------------------------------------------------------------- |
| `AUTH_SECRET` | A stable random value of at least 32 characters. Generate with `openssl rand -hex 32`. |
| `AUTH_BASE_URL` | The public HTTPS backend origin users and providers can reach. |
| `AUTH_ALLOWED_ORIGINS` | Optional comma-separated extra browser origins. Keep this list narrow. |
| `SMTP_CONNECTION_URL` | Required for email verification, password reset, and invitation email delivery, unless `EMAIL_TRANSPORT=ses`. |
| `EMAIL_FROM_ADDRESS` | The sender address for transactional email. |
`AUTH_SECRET` protects sessions and encrypted auth material. Do not generate a
new value on every deploy. A rotation can invalidate sessions. It can also
prevent ctxpipe from reading the encrypted signing keys that you have. Plan each rotation
as an operational change.
Open the product through `AUTH_BASE_URL`. Do not send browser users directly
to the private UI service. Provider callbacks, MCP OAuth, cookies, and the API
need one consistent origin.
## Email and password
Email/password sign-in is available by default. Set up SMTP before you invite a
team. Then verification, invitation, and password-reset messages leave the
deployment reliably.
When you set SMTP and `EMAIL_FROM_ADDRESS`, a new email/password account must
verify its address before it can sign in. ctxpipe sends a fresh link to an
unverified account that signs in. A password reset also verifies the address. It
signs out the other sessions of the account. Without SMTP, ctxpipe only logs the
messages. Verification is then not required, and the backend logs a warning at
startup.
## Social providers (OAuth)
ctxpipe supports GitHub, Google, and Microsoft social sign-in.
Each provider is on only when you set its client ID and secret in the backend
environment. Then the sign-in option shows.
| Provider | Env vars required |
| --------- | ------------------------------------------------ |
| GitHub | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` |
| Google | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` |
| Microsoft | `MICROSOFT_CLIENT_ID`, `MICROSOFT_CLIENT_SECRET` |
GitHub social sign-in credentials are separate from the
[GitHub App](/docs/self-hosting/github) that your deployment owns. That App
gives access to repositories and writes connector content.
Register each provider callback under `AUTH_BASE_URL`. Test sign-in through the
same public origin that users use in production.
## Passkeys and two-factor authentication
Passkeys are available on compatible browsers and platforms. Users can also
enable TOTP two-factor authentication with a standard authenticator app. Verify
both through your production TLS domain. WebAuthn credentials belong to one
origin.
## MCP OAuth and API keys
ctxpipe is an OAuth provider for downstream MCP clients. It supports discovery,
dynamic client registration, and device authorization for compatible headless
clients.
Give the client this URL:
```text
{AUTH_BASE_URL}/mcp
```
Each OAuth grant holds a set of organizations. The user selects the organizations
on the consent screen, and the user can change the set later on the
**Connected apps** page (`/.auth/account/connected-apps`). Each tool call names
one organization with the `org` argument. The server allows the call only when
the organization is in the grant set and the user is still a member of it.
For the old `?orgSlug=` URL parameter, see [MCP](/docs/self-hosting/mcp).
OAuth is the normal MCP flow for people. The consent screen grants
`ctxpipe:read` and `offline_access`. It also offers **Also allow changes**
(`ctxpipe:write`) when a client asks for it. The MCP tools need only
`ctxpipe:read`. API keys never hold `ctxpipe:write`.
| Token | Lifetime |
| --------------------------------------- | ---------------------------------------------------------- |
| Browser session | 30 days |
| OAuth access token | 4 hours, renewed with the refresh token (`offline_access`) |
| OAuth access token with `ctxpipe:write` | 1 hour |
API keys are stored in the database of this deployment, so there is no
operator secret to configure. For personal and organization keys, the headers,
and the client setup, see [API keys](/docs/mcp/api-keys). When discovery or
authorization fails, see [MCP diagnostics](/docs/self-hosting/mcp).
## Production checks
* `AUTH_SECRET` is stable across backend instances and deploys.
* `AUTH_BASE_URL` exactly matches the browser origin.
* Provider callbacks use HTTPS and the same host.
* Extra trusted origins are explicit rather than wildcarded.
* SMTP can deliver invitation and reset messages.
* Passkey registration and TOTP sign-in work on the production domain.
* An MCP client can discover OAuth and complete authorization.
# Configuration (https://docs.ctxpipe.ai/docs/self-hosting/configuration)
Environment variables control the configuration. Set only the values that your
deployment needs. Treat the required auth, database, model, and public-origin
values as production secrets.
## What a deployment needs
* A [deployment path](/docs/self-hosting#choose-a-deployment-path): Docker Compose or AWS CDK.
* PostgreSQL with pgvector. Compose uses PostgreSQL 17. AWS CDK
uses Aurora PostgreSQL 16.
* A graph database. FalkorDB is in the Compose stack.
* ClickHouse 26.10 for the new knowledge store. ClickHouse is in the Compose
stack. AWS CDK does not deploy it yet.
* A model provider: OpenRouter, Azure OpenAI, Amazon Bedrock, or another
OpenAI-compatible API.
* A public HTTPS origin for auth callbacks, webhooks, MCP, and Forge.
* Optional: an SMTP provider for email.
On Docker Compose, set the values in `.env`. Compose passes only some of them
to the services. See
[Settings that need an override file](/docs/self-hosting/deployment/docker#settings-that-need-an-override-file).
## Required variables
The backend does not start without these:
| Variable | Description |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_URL` | Runtime PostgreSQL URL for backend, worker, and codesearch. Use the `ctxpipe_app` role, which has no `BYPASSRLS`. Example: `postgresql://ctxpipe_app:pass@host:5432/db`. `migrate` uses the table-owner URL. Compose and AWS CDK set both URLs. |
| `AUTH_SECRET` | A random string of 32 or more characters, for session signing. Generate it with `openssl rand -hex 32`. Keep it stable. A new value signs everybody out and, without `CONNECTION_SECRETS_ENCRYPTION_KEY`, makes the stored connector secrets unreadable. |
Set these too for a deployment that people use:
| Variable | Description |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `AUTH_BASE_URL` | The public URL of your deployment, for example `https://ctx.example.com`. The default, `https://localhost:3000`, works only on your machine. |
| `CTXPIPE_PUBLIC_APP_URL` | The public backend URL in the UI image, at build time. Usually the same as `AUTH_BASE_URL`. Compose and AWS CDK read it. |
| `MODEL_PROVIDER_API_KEY` | The API key of your model provider. Required for each provider except `bedrock`. |
| `CODESEARCH_URL` | The URL of codesearch. Repository search and indexing need it. The backend and the worker must both reach it. Compose and AWS CDK set it. |
## Email
Email is optional. Without both values, ctxpipe turns off email verification
and sends no password reset or invitation email. The backend logs a warning.
| Variable | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SMTP_CONNECTION_URL` | SMTP connection URL, for example `smtps://user:pass@smtp.resend.com:465` |
| `EMAIL_FROM_ADDRESS` | From address for transactional email, for example `noreply@example.com` |
| `EMAIL_TRANSPORT` | `smtp` (the default) or `ses`. With `ses`, the backend sends through the Amazon SES v2 API. It uses the AWS credentials of its process (for example an ECS task role) and its `AWS_REGION`. It needs no `SMTP_CONNECTION_URL`. The role needs `ses:SendEmail` on the sender identity. |
## Social sign-in
All social providers are optional. Omit the variables to disable that provider.
| Variable | Description |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITHUB_CLIENT_ID` | Client ID of the GitHub App (sign-in, and the check that a user can access an installation) |
| `GITHUB_CLIENT_SECRET` | Client secret of the GitHub App |
| `GITHUB_WEB_URL` | Optional. The base URL, in place of `https://github.com`, of the GitHub pages that the browser opens for the deployment's App (the authorize page and the install page), when a forwarder serves them. Set it together with `GITHUB_REDIRECT_URI`. Default: `https://github.com` |
| `GITHUB_REDIRECT_URI` | Optional. The callback URL that the GitHub App lists, when a forwarder serves it. Set it together with `GITHUB_WEB_URL`. Default: `{AUTH_BASE_URL}/.auth/api/v1/auth/callback/github` |
| `GOOGLE_CLIENT_ID` | Google OAuth client ID |
| `GOOGLE_CLIENT_SECRET` | Google OAuth client secret |
| `MICROSOFT_CLIENT_ID` | Microsoft Entra application client ID |
| `MICROSOFT_CLIENT_SECRET` | Microsoft Entra client secret |
See [Authentication](/docs/self-hosting/authentication) for OAuth and session
behavior.
## Models and embeddings
| Variable | Description |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MODEL_PROVIDER` | Optional. `openai-like` default, `openrouter`, `azure`, or `bedrock`. |
| `MODEL_PROVIDER_API_KEY` | API key for OpenAI-compatible providers. Omit for native Bedrock IAM. |
| `MODEL_PROVIDER_URL` | Optional for `openai-like` and `openrouter`; required for Azure; not used by native Bedrock. |
| `MODEL_BEDROCK_AWS_REGION` | Optional explicit region for native Bedrock; otherwise use the AWS runtime region. |
| `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, `MODEL_HIGH_NAME` | Optional model names per tier. |
| `MODEL_EMBEDDING_NAME` | Optional embedding model ID. |
| `MODEL_VISION_NAME` | Optional. The one OpenRouter model that reads images for the knowledge plane (default `openai/gpt-6.1-terra`). It must take image input and have a ZDR endpoint on OpenRouter. |
See [Model configuration](/docs/self-hosting/models) for provider-specific
examples and embedding requirements.
## Graph database
| Variable | Description |
| ------------------------------------ | ------------------------------------------------------------------------------------------------ |
| `GRAPH_DB_PROVIDER` | Optional. `falkordb` (default), `neo4j-enterprise`, `neo4j-community`, `memgraph`, or `neptune`. |
| `GRAPH_DB_URI` | FalkorDB Redis URI or Bolt URI for another supported graph database. |
| `GRAPH_DB_USER`, `GRAPH_DB_PASSWORD` | Optional credentials for providers that require them. |
See [Graph databases](/docs/self-hosting/graph-databases) for provider support
and tenancy behavior.
## Knowledge store
ClickHouse holds the content of the new knowledge store: the documents, their
sections, and the relations between them. ctxpipe makes this content from the
files of each Workspace repository.
The new knowledge store is off for every organization. A later release
explains how to turn it on.
| Variable | Description |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CLICKHOUSE_URL` | Required for an organization on the new knowledge store. The ClickHouse HTTP URL with the user, the password, and the database: `http://user:password@host:8123/database`. URL-encode special characters in the password. Each process gets a different user (see below). |
| `CLICKHOUSE_READER_PASSWORD` | Migrations only. The password of the user `ctxpipe_reader`. |
| `CLICKHOUSE_WRITER_PASSWORD` | Migrations only. The password of the user `ctxpipe_writer`. |
Give each process its own ClickHouse user in `CLICKHOUSE_URL`:
* **Migrations:** an admin user with SQL access management
(`access_management`). The migrations create the tables, a row policy for
each table, and the two users below. Give the admin user to the migrations
only.
* **Backend:** `ctxpipe_reader`. It can only read the knowledge tables.
* **Worker:** `ctxpipe_writer`. It can read, insert, and delete rows in the
knowledge tables. It cannot manage users, grants, or row policies.
The migrations create or update a user only when its password is set. Without
the password, the user does not exist and cannot connect. The migrations refuse
the same value for two of the admin, reader, and writer passwords. They also
refuse the reader and writer passwords when the admin password is the Compose
default `ctxpipe`. They store a salted SHA-256 hash of the password. To change
a password, set the new value and run the migrations again. If
`CLICKHOUSE_URL` is not set, the migrations skip ClickHouse.
The knowledge tables move a part to the volume `cold` when its newest row is
30 days old. Thus the `default` storage policy of the server must have the
volumes `hot` and `cold`, or the migrations fail. Compose loads
[`clickhouse-storage.xml`](https://github.com/ctxpipe-ai/ctxpipe/blob/main/apps/backend/src/knowledge-store/clickhouse-storage.xml)
into `/etc/clickhouse-server/config.d/`. On another ClickHouse server, add that
file in the same folder.
The Compose service keeps both volumes on its local disk.
### Keep old parts on S3
This example replaces the disk of the volume `cold` with an S3 bucket behind a
cache disk. Change the bucket URL, the Region, and the cache size:
```xml title="clickhouse-storage-s3.xml"
s3
https://example-bucket.s3.us-west-2.amazonaws.com/clickhouse/
true
cache
cold_s3
cold
50Gi
131072
default
cold
0
0
```
On Compose, mount your file in place of the default file, in a
`docker-compose.override.yml`:
```yaml title="docker-compose.override.yml"
services:
clickhouse:
volumes:
- ./clickhouse-storage-s3.xml:/etc/clickhouse-server/config.d/storage.xml:ro
environment:
AWS_ACCESS_KEY_ID: ${CLICKHOUSE_S3_ACCESS_KEY_ID}
AWS_SECRET_ACCESS_KEY: ${CLICKHOUSE_S3_SECRET_ACCESS_KEY}
```
The search reads small ranges of the vector column, so the example sets
`boundary_alignment` of the cache disk to `131072` (128 KiB). With the default
alignment of 4 MiB, a search that is not in the cache downloads about 1 GiB
from S3 for 46 MiB of data on 1 million sections.
The cache files go to `/var/lib/clickhouse/caches/cold/`. Do not put the cache
path under `/var/lib/clickhouse/disks/`: the image then makes that folder as
`root`, and ClickHouse cannot start.
With `use_environment_credentials`, ClickHouse uses the AWS credentials of its
environment, or the instance role on EC2. Give them read, write, list, and
delete access to the bucket prefix only. Set the S3 disk before the first
start. A part that is already on the local volume `cold` does not move to S3.
### Recover the knowledge store
ClickHouse holds only content that ctxpipe makes from the Workspace
repositories. If you lose the ClickHouse data:
1. Restore a ClickHouse backup, if you have one.
2. Run the migrations again, also after a restore. They make each missing
table and the users `ctxpipe_reader` and `ctxpipe_writer`. On Compose, set
the reader and writer passwords in `.env`, then run:
```bash
docker compose --profile deploy run --rm migrate
```
3. Delete the index state of the knowledge store in Postgres. Do this also
after a restore: the backup can be older than the index state, and the
next run then does not apply the commits between them.
```bash
docker compose exec postgres psql -U ctxpipe -c "DELETE FROM knowledge_state"
```
The next hydrate of each Workspace is a full rebuild. Until a Workspace
completes its full rebuild, the graph database answers for it.
## Chat sandboxes
Workspace chat runs each conversation's agent in a sandbox. Docker Compose and
AWS CDK set these values for you. On other platforms, set them on the backend and
the worker.
| Variable | Description |
| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SANDBOX_PROVIDER` | `docker`, `agentcore`, or `unsandboxed`. Set it to `docker` on a self-hosted deployment; Docker Compose and AWS CDK set it. Without a value, chat uses Docker, but the plain git rules then use the strict hosted rule: they use the self-host rule only when the value is exactly `docker` (see below). If the Docker daemon does not answer, chat returns 503. The backend and worker images refuse `unsandboxed`: it is only for local development. |
| `DOCKER_HOST` | Docker API address, for example `tcp://dind:2376`. If you do not set it, the backend uses the local Docker socket. |
| `DOCKER_TLS_VERIFY`, `DOCKER_CERT_PATH` | Mutual TLS to the Docker API. Set `DOCKER_TLS_VERIFY=1`. `DOCKER_CERT_PATH` is a directory with `ca.pem`, `cert.pem`, and `key.pem`. |
| `SANDBOX_CHAT_IMAGE` | Chat sandbox image on that daemon. The default is `ctxpipe-chat-sandbox:opencode-`, which is in no registry. On other platforms, build `scripts/chat-sandbox` on that daemon, or use `ghcr.io/ctxpipe-ai/chat-sandbox:` once that package is public. Compose builds `ctxpipe-chat-sandbox:local`. AWS CDK uses `ghcr.io/ctxpipe-ai/chat-sandbox` at the commit SHA tag that the package pins. |
| `SANDBOX_CALLBACK_HOST` | Optional. Host name that sandboxes use to send requests to the backend, through Agent Vault. It must be a host name with a dot (for example `backend.sandbox.ctxpipe.internal`), not an IP address: Agent Vault rules take host names. If you do not set it and `DOCKER_HOST` is remote, the backend uses its own address on the route to the Docker daemon, which works only with `SANDBOX_CALLBACK_DNS_SUFFIX`. Compose sets it. |
| `SANDBOX_CALLBACK_DNS_SUFFIX` | Optional. With it, that derived address becomes the name `ip----.`, for example the VPC DNS name on AWS. AWS CDK sets it. |
| `AGENT_VAULT_ADDR` | Required for Docker sandboxes. The Agent Vault API, for example `http://agent-vault:14321`. Without it, or when Agent Vault does not answer, Docker chat returns 503. Compose and AWS CDK set it. |
| `AGENT_VAULT_OWNER_PASSWORD`, `AGENT_VAULT_OWNER_PASSWORD_FILE` | The Agent Vault owner password, as a value or as a file. Compose and AWS CDK generate it; do not set it there. On other platforms, generate one and keep it: the backend registers it on its first login, and an Agent Vault with an owner refuses another password. |
| `AGENT_VAULT_SANDBOX_PROXY_HOST` | Optional. The host name that sandboxes use to connect to the Agent Vault proxy, when sandboxes cannot use the host of `AGENT_VAULT_ADDR`. Give only a host name: no scheme, no port, and no path. The port is the proxy port that Agent Vault reports (`14322`). Hosted ctxpipe on AWS sets it to the DNS name of the Agent Vault endpoint in the sandbox VPC, because AgentCore sandboxes cannot connect to the private name of the API. That endpoint must forward only the proxy port, never the API port (`14321`): code in a sandbox must not get access to the Agent Vault API. If you do not set it, sandboxes use the address of the `AGENT_VAULT_ADDR` host. Docker Compose and AWS CDK do not set it. |
| `AGENTCORE_RUNTIME_ARN`, `AGENTCORE_RUNTIME_QUALIFIER` | Required when `SANDBOX_PROVIDER=agentcore`. Hosted ctxpipe uses this provider; AWS CDK does not support it yet. The ARN of the Amazon Bedrock AgentCore runtime that runs the `agentcore` target of the chat sandbox image, and its endpoint name (default `DEFAULT`). The backend also needs Agent Vault, `AGENT_VAULT_SANDBOX_PROXY_HOST` when the sandboxes are in a different VPC, and `SANDBOX_CALLBACK_DNS_SUFFIX` or `SANDBOX_CALLBACK_HOST`. Each process in the sandbox can read the credentials of the runtime's execution role at `169.254.169.254`. Thus that role must allow only the pull from the sandbox image's repository. It must also allow log group creation and log writes, only on the runtime's own log group. It must refuse each request that does not come through the sandbox VPC endpoints. AgentCore's own `BedrockAgentCore-*` sessions are the exception: AgentCore creates the log group from its own network. The ingest check allows that address because of this rule. |
### Agent Vault rules on AgentCore
With `SANDBOX_PROVIDER=agentcore`, the Agent Vault network allowlist
(`AGENT_VAULT_NETWORK_ALLOWLIST`) holds the backend's own subnets. Thus a
sandbox can send requests to the backend task that runs its turn: to the
model proxy on `PORT`, and to the tool bridge on a port that the kernel
selects. Agent Vault has one allowlist for all its rules. It resolves the
host of a rule again each time it connects. Thus the DNS of a host that a
customer names, for example a plain git repository, can point at a backend
task when Agent Vault connects. This is also possible when the host
resolved to a public address at the time when the backend wrote the rule.
The backend uses the port to keep these rules away from the backend's ports:
* Only a callback rule can use a port other than 443. A callback rule is a
model proxy or tool bridge rule that the backend makes, and its host is
the backend's own callback name: `SANDBOX_CALLBACK_HOST`, or
`ip----.` for an address of the
backend. Each other rule gets port 443.
* The backend refuses each other rule that names another port, for example
`PORT`, a tool bridge port, or 80 (plain HTTP).
* Agent Vault forwards a request only when a rule matches its host and its
port, and it connects to that port. A DNS answer cannot change the port.
* Each run vault refuses each host that no rule names.
* When the backend writes a rule, it refuses a host that resolves to an
address that is not public. If the host does not resolve, the sandbox
does not start. This check is only a first layer, because the DNS answer
can change after the check.
The limit: a host that a customer names can still connect to port 443 of
each address in the allowlist. Do not run a service that listens on port
443 in the backend's subnets, and do not add other subnets to the
allowlist. Agent Vault 0.40.2 has no network allowlist for each rule, and no
way to pin a rule to the addresses that the backend checked. With the other
values of `SANDBOX_PROVIDER`, the backend writes the rules as they are.
### Plain git repositories
A plain git repository is a repository that ctxpipe reads with no credential
(see [Linked repositories](/docs/workspaces/linked-repositories#plain-git-repositories)).
With `SANDBOX_PROVIDER=docker`, its URL can use `https://` or `http://`,
another port, an IP address, or an internal host name. ssh (`git@…` or
`ssh://`), `git://`, and `file://` URLs are not supported. ctxpipe never reads a
host that resolves to a cloud metadata or link-local address
(`169.254.0.0/16`, `fe80::/10`, or `fd00:ec2::254`). With each other value
of `SANDBOX_PROVIDER`, or with no value, the URL must use `https://` on the
default port, and each address of its host must be public. The backend and
codesearch read the repository through the proxy of their environment
(`HTTPS_PROXY`, `NO_PROXY`), if you set one.
| Variable | Description |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENT_VAULT_NETWORK_ALLOWLIST` | Docker Compose only (`.env`). Optional. The private addresses and CIDRs that Agent Vault may connect to, separated by commas. Compose always allows `172.30.99.10`, the backend, and adds your entries after it. To ingest a plain git repository on a private git server, add only the single host address of your git server, for example `10.20.0.15`. Without it, the ingestion sandbox cannot fetch the repository. Chat-sandbox vaults forward each host that no rule matches. An address or CIDR in this list therefore lets every chat agent reach every port and path at that address. Add only the single host address of the git server, never a whole subnet. Never add a range that covers `172.30.99.0/24`, and never add a link-local (`169.254.0.0/16`), loopback, or cloud metadata range. On AWS CDK, use `sandboxHost.agentVaultNetworkAllowlist` (see [AWS](/docs/self-hosting/deployment/aws#chat-sandbox-host)). |
## GitHub App connector
Repository connectors use a GitHub App that your deployment owns. The in-product
wizard stores the ID, the PEM private key, and the webhook secret of the App,
encrypted, for the connection.
| Variable | Description |
| ----------------------------------- | --------------------------------------------------------------------------------- |
| `GITHUB_APP_ID` | Optional after migrations. GitHub App numeric ID. |
| `GITHUB_PRIVATE_KEY` | Optional after migrations. PEM private key for the App. |
| `GITHUB_WEBHOOK_SECRET` | Optional after migrations. Also used for the legacy webhook route. |
| `GITHUB_APP_SLUG` | Optional. Public app slug used for default install URLs. |
| `CONNECTION_SECRETS_ENCRYPTION_KEY` | Optional 64-character hex key. When unset, encryption derives from `AUTH_SECRET`. |
The wizard reserves the connection-specific webhook URL and gives the exact
value to paste into GitHub.
`GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` are for signing in with GitHub
and for the one-time authorization that proves access to an installation.
Use the client ID and secret of the App in `GITHUB_APP_ID`. GitHub App
credentials are for repository installation and webhook access.
See [GitHub](/docs/self-hosting/github) for App registration, permissions,
events, credentials, installation, and end-to-end verification.
## Atlassian and Confluence
Self-hosted Confluence uses the same product wizard as hosted ctxpipe. But
your deployment or organization must first provide the Atlassian and Forge
settings.
| Variable | Description |
| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `ATLASSIAN_CLIENT_ID`, `ATLASSIAN_CLIENT_SECRET` | Optional global Atlassian 3LO app for account linking. If unset, each Forge connection can store its own 3LO credentials through the wizard. |
| `CONFLUENCE_FORGE_INSTALL_URL` | Optional deployment-wide Forge install URL fallback when a Forge connection has no install URL in `connections.config`. |
See [Confluence & Atlassian](/docs/self-hosting/confluence-atlassian) for the
full operator flow.
## Slack
Self-hosted Slack uses one Slack app that your deployment owns. Organizations
authorize that app. They do not create per-tenant Slack apps or Event
Subscription URLs.
| Variable | Description |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `SLACK_CLIENT_ID`, `SLACK_CLIENT_SECRET` | Slack app OAuth credentials. Required to enable the Slack connector. |
| `SLACK_SIGNING_SECRET` | Verifies Event Subscription requests (`X-Slack-Signature`). |
| `SLACK_REDIRECT_URI` | Optional. Override when the public OAuth callback differs from `{AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback`. |
Set the client ID, client secret, and signing secret together. Then restart or
redeploy the backend. The backend and the worker must share the same
`AUTH_SECRET` and, when configured, `CONNECTION_SECRETS_ENCRYPTION_KEY`. Then
the background capture jobs can use the encrypted Slack bot tokens.
Set the Slack Event Subscriptions Request URL to:
```text
https:///api/v1/webhook/slack
```
See [Slack](/docs/self-hosting/slack) for scopes, `app_mention` Events, and
intent-based thread capture.
## Linear
ctxpipe can store the Linear OAuth application and the webhook signing secret
on the connection (Register in the product UI). `LINEAR_*` is an optional
deployment-wide fallback.
| Variable | Description |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------- |
| `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET` | Optional shared Linear OAuth application credentials. When both are set, the wizard skips Register. |
| `LINEAR_REDIRECT_URI` | Optional exact callback override. Defaults to `{AUTH_BASE_URL}/api/v1/integrations/linear/callback`. |
| `LINEAR_WEBHOOK_SECRET` | Optional shared signing secret for `/api/v1/webhook/linear`. Per-connection secrets are stored on the row. |
See [Linear](/docs/self-hosting/linear) for application registration, webhook
subscriptions, preview limitations, and troubleshooting.
## Notion
Notion uses one public integration and one webhook subscription. Your
deployment owns both.
| Variable | Description |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| `NOTION_CLIENT_ID`, `NOTION_CLIENT_SECRET` | Public integration OAuth credentials. Both are required to enable the connector. |
| `NOTION_REDIRECT_URI` | Optional exact callback override. Defaults to `{AUTH_BASE_URL}/api/v1/connectors/notion/oauth/callback`. |
| `NOTION_WEBHOOK_SECRET` | Verification token shown by Notion after the first webhook delivery. Required for signed updates. |
Put the OAuth credentials on every service that refreshes tokens or does
connector work. Then register the single deployment webhook.
See [Notion](/docs/self-hosting/notion) for provisioning and verification.
## PagerDuty
PagerDuty can use an OAuth application that your deployment owns. After a user
authorizes an account, ctxpipe creates the signed webhook subscription. These
variables are an optional shortcut, as on hosted ctxpipe. A self-hosted
deployment can instead save the app on the connection in the setup wizard.
| Variable | Description |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `PAGERDUTY_CLIENT_ID`, `PAGERDUTY_CLIENT_SECRET` | Optional shared PagerDuty OAuth application. When unset, save client id and secret on the connection in the wizard. |
| `PAGERDUTY_REDIRECT_URI` | Optional exact callback override. Defaults to `{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback`. |
There is no `PAGERDUTY_WEBHOOK_SECRET`. ctxpipe stores the signing secret of
each subscription encrypted on the connection.
See [PagerDuty](/docs/self-hosting/pagerduty) for application registration and
verification.
## Observability
Telemetry is off until you set it. ctxpipe does not ship a collector or
ClickStack in the Compose `deploy` profile or in `@ctxpipe/aws-cdk`. Unset
values keep logs on stdout and do not start trace or metric export. Empty
values count as unset.
Point the signal URLs at your own OTLP/HTTP backend. Export starts from these
variables. Include the signal path on each URL.
```bash
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otel.example.com/v1/traces
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://otel.example.com/v1/logs
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://otel.example.com/v1/metrics
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer xxx
```
| Variable | Description |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Traces URL, including `/v1/traces`. Trace export stays off when this is unset. |
| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Logs URL, including `/v1/logs`. Unset keeps logs on stdout. |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Metrics URL, including `/v1/metrics`. |
| `OTEL_EXPORTER_OTLP_HEADERS` | Optional headers, for example `Authorization=Bearer xxx`. |
| `OTEL_SERVICE_NAME` | Resource `service.name`. Compose and CDK set `backend`, `openworkflow`, `ui`, and `codesearch` per container. Compose does not apply a shared name from the root `.env`. |
| `OTEL_RESOURCE_ATTRIBUTES` | Extra resource attributes, for example `deployment.environment=production`. |
The `deployment.environment` value applies to backend traces, metrics, and logs,
to codesearch, and to the UI relay. It is
`DEPLOYMENT_ENVIRONMENT` when that is set. Hosted ctxpipe sets this variable;
a self-hosted install leaves it unset, because a `pr-N` value turns on preview
behavior (no email verification). Otherwise it is
`deployment.environment` from `OTEL_RESOURCE_ATTRIBUTES`. Otherwise it is
`production` when `NODE_ENV=production`. Otherwise it is `development`. Backend
and codesearch also set `service.namespace=ctxpipe`. Codesearch always exports `service.name`
`codesearch`.
LLM spans are OpenTelemetry spans with the GenAI semantic conventions:
`gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`,
`gen_ai.usage.output_tokens`, and `gen_ai.usage.cost` (USD, when the model
gateway reports it). A span also holds the prompt and the output text in
`gen_ai.input.messages` and `gen_ai.output.messages`. The backend logs never
hold them. `@ctxpipe/aws-cdk` takes one base `otel.endpoint`
and appends `/v1/traces`, `/v1/logs`, and `/v1/metrics`. See
[AWS](/docs/self-hosting/deployment/aws).
### Browser telemetry
The UI enables browser telemetry only when
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` is set on the UI server. The browser posts
traces to the same origin at `/.otel/v1/traces`. The relay also accepts
`/.otel/v1/logs`, but the SDK does not use it. Replay is off, and the SDK
emits console capture as spans. The UI server forwards traces to
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and logs to
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, using `OTEL_EXPORTER_OTLP_HEADERS`. When
the traces endpoint is unset, the browser SDK does not start. A signal whose
endpoint is unset returns `204`.
## Advanced
| Variable | Description |
| --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AUTH_ISSUER` | Override the OAuth issuer claim. Defaults to `AUTH_BASE_URL`. |
| `AUTH_ALLOWED_ORIGINS` | Comma-separated list of allowed CORS origins for auth endpoints. |
| `OPENWORKFLOW_CONCURRENCY` | In-flight OpenWorkflow runs per worker process. Default `4` when unset. Compose sets `6`. AWS CDK sets a value for each size. |
| `AUTH_TOKEN_AUDIENCE_CODESEARCH` | Optional. The audience of the tokens that the backend sends to codesearch. If you set it, set the same value on the backend, the worker, and codesearch. |
| `CODESEARCH_INDEXER_CONCURRENCY` | Max concurrent Zoekt/SCIP child processes on codesearch, and SCIP HTTP batch size on the worker. Default `2`. At most one of them runs a SCIP indexer at a time. |
| `CODESEARCH_INDEX_PIPELINE_CONCURRENCY` | Max distinct repos with in-flight OpenWorkflow index-phase HTTP (clone included). Default `2`. Codesearch returns 429 when full. |
## Related pages
# Graph databases (https://docs.ctxpipe.ai/docs/self-hosting/graph-databases)
When you self-host this service, you must set up an OpenCypher-compatible graph database for knowledge graphs, RAG, and related workloads. This guide helps you choose and set up one of the supported engines.
## Quick Start (FalkorDB)
The easiest way to start is **FalkorDB**. The default stack includes it as a Docker container. You do not need extra setup. Run:
```bash
docker compose --profile deploy up -d
```
By default, the backend connects to FalkorDB with the **native driver** at `redis://falkordb:6379`. For host-based deployments, point `GRAPH_DB_URI` at your FalkorDB instance (for example `redis://localhost:6379`). Or use any OpenCypher-compatible database with the Bolt driver.
On Docker Compose, the `GRAPH_DB_*` values in `.env` do not reach the backend.
Set them in a `docker-compose.override.yml`. See
[Settings that need an override file](/docs/self-hosting/deployment/docker#settings-that-need-an-override-file).
## Supported Engines
You can use any of these OpenCypher-compatible engines. FalkorDB uses its native driver (Redis protocol). Others use Bolt.
| Engine | Best for | Notes |
| -------------------- | ----------------------------- | ------------------------------------------------------------ |
| **FalkorDB** | Easy start, multi-tenant SaaS | Open-source. Included in Docker Compose. Graphs auto-create. |
| **Neo4j Enterprise** | Enterprise multi-tenant | Requires `CREATE DATABASE` per org (admin). |
| **Neo4j Community** | Cost-conscious multi-tenant | One database per instance. Use instance-per-tenant. |
| **Memgraph** | Enterprise, real-time | Enterprise edition for multi-database. |
| **Neptune** | AWS-hosted | Managed service. Single graph per cluster. |
Set `GRAPH_DB_PROVIDER` to match your engine. The service infers the tenancy mode automatically.
## Environment Variables
| Variable | Required | Default | Description |
| ------------------------ | ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------- |
| `GRAPH_DB_URI` | No | `redis://falkordb:6379` | FalkorDB: `redis://host:6379`. Others: Bolt URI (e.g. `bolt://host:7687`). |
| `GRAPH_DB_USER` | No | `default` / `neo4j` | Username for auth. |
| `GRAPH_DB_PASSWORD` | No | `""` | Password for auth. |
| `GRAPH_DB_PROVIDER` | No | `falkordb` | Engine: `falkordb`, `neo4j-enterprise`, `neo4j-community`, `memgraph`, `neptune`. |
| `GRAPH_DB_URI_` | Yes (instance-per-tenant) | - | Per-org URI. Required when using Neo4j Community or Neptune. No fallback to `GRAPH_DB_URI`. |
**Example (FalkorDB, Docker network):**
```bash
GRAPH_DB_URI=redis://falkordb:6379
GRAPH_DB_PROVIDER=falkordb
```
**Example (Neo4j Community, instance-per-tenant, org slugs `acme`, `piedpiper`):**
```bash
GRAPH_DB_URI=bolt://neo4j.example.com:7687
GRAPH_DB_URI_acme=bolt://neo4j-acme.example.com:7687
GRAPH_DB_URI_piedpiper=bolt://neo4j-piedpiper.example.com:7687
GRAPH_DB_USER=neo4j
GRAPH_DB_PASSWORD=yourpassword
GRAPH_DB_PROVIDER=neo4j-community
```
**Example (Neptune):**
```bash
GRAPH_DB_URI_acme=bolt+s://your-cluster.region.neptune.amazonaws.com:8182
GRAPH_DB_PROVIDER=neptune
```
## Tenancy Modes
All setups are **multi-tenant**. The service must isolate the graph data of each organization from the data of the others. The service infers the tenancy strategy from the provider in `GRAPH_DB_PROVIDER`.
### Database-per-tenant (shared instance)
**Providers**: FalkorDB, Neo4j Enterprise, Memgraph
These engines support **multiple graphs or databases in one instance**, with strong isolation between them. One shared instance serves many organizations. Each organization gets its own graph or database (for example `org_abc123`). The service passes the database name of the organization when it runs queries. Data therefore stays isolated at the database layer.
### Instance-per-tenant (separate instance per org)
**Providers**: Neo4j Community, Neptune
Neo4j Community and Neptune **do not support multiple databases or graphs in one instance**. Each instance has one logical graph. For per-organization isolation, the service uses **instance-per-tenant**. Each organization must have its own database instance. The service routes each query to the instance of the organization. The deployment topology enforces isolation, not the database engine.
Set `GRAPH_DB_URI_` for each organization (for example `GRAPH_DB_URI_acme`, `GRAPH_DB_URI_piedpiper`). There is no fallback to `GRAPH_DB_URI`. Each organization must have its own URI. The AWS CDK construct sets this variable for its one `orgSlug`, so CDK users need to do nothing.
## Engine-Specific Setup
### FalkorDB
* **Docker**: `falkordb/falkordb`. Uses Redis protocol on port 6379 by default. Pin a version in production; hosted ctxpipe pins one.
* **URI**: `redis://host:6379` (native driver, recommended for production).
* **User**: `default`
* **Password**: Empty by default. With Redis `--requirepass`, use that password.
### Neo4j
* **URI**: `bolt://host:7687`
* **User**: `neo4j`
* **Password**: Set via `GRAPH_DB_PASSWORD` or `NEO4J_AUTH` (Docker).
### Memgraph
* **URI**: `bolt://host:7687`
* **User**: Typically `memgraph` or as per your deployment.
* **Enterprise**: Use `memgraph` provider for database-per-tenant.
### Neptune
* **URI**: The Bolt endpoint of your cluster, with TLS (e.g. `bolt+s://cluster.region.neptune.amazonaws.com:8182`).
* **Auth**: ctxpipe does not support IAM database authentication. Turn IAM auth off on the cluster, as the AWS CDK construct does. Set `GRAPH_DB_USER` and `GRAPH_DB_PASSWORD` only if your cluster needs them. See [AWS Neptune docs](https://docs.aws.amazon.com/neptune/).
# MCP (https://docs.ctxpipe.ai/docs/self-hosting/mcp)
Self-hosted ctxpipe exposes the same organization-scoped Streamable HTTP endpoint as
hosted ctxpipe:
```text
{AUTH_BASE_URL}/mcp
```
Every client uses this one URL. During authorization, the user chooses the
organizations that the client can read. Each tool call names one organization in
the `org` argument. The server accepts the old `?orgSlug=` parameter and ignores
it.
`AUTH_BASE_URL` must be the public origin clients use, including TLS. Client
JSON shapes and OAuth install paths are the same as on hosted ctxpipe. See
[Connect your agent](/docs/mcp/mcp-docs) and
[Authentication](/docs/self-hosting/authentication).
This page is for operators. Confirm that the endpoint is reachable. Then debug
OAuth and tools on a **test organization**.
`npx ctxpipe doctor mcp` checks HTTP routing, TLS or reachability, the
unauthenticated Bearer challenge, and OAuth discovery. A `ready-for-oauth`
result does **not** prove that browser OAuth, authenticated `tools/list`, or
an MCP tool call work. The command does not accept OAuth bearer tokens and
does not test STDIO servers. When `CTXPIPE_API_KEY` is set, it sends the key
as `x-api-key` and checks an authenticated initialize.
## Connect a client
Use the same config as [Quickstart](/docs/self-hosting/quickstart), with your
deployment origin:
```json
{
"mcpServers": {
"ctxpipe": {
"type": "http",
"url": "https://ctx.example.com/mcp"
}
}
}
```
The MCP client must complete OAuth against this deployment. CLI setup sign-in
and MCP client OAuth are separate flows.
## Diagnose discovery
Run the lightweight doctor against the public MCP URL:
```bash
npx ctxpipe doctor mcp \
--url "$AUTH_BASE_URL/mcp"
npx ctxpipe doctor mcp \
--url "$AUTH_BASE_URL/mcp" \
--json
```
`--json` is machine-readable. Failures set a non-zero exit code. Loopback HTTP
works with a warning. Remote endpoints must use HTTPS.
| Check | Passes when |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Target | Path is `/mcp`; remote origins use HTTPS |
| Backend status | `GET /.status` returns JSON with `status: "ok"` |
| OAuth challenge | Unauthenticated Streamable HTTP `initialize` returns `401` with a Bearer `WWW-Authenticate` challenge that includes `resource_metadata` |
| Protected resource | RFC 9728 metadata matches the `/mcp` audience and names an authorization server |
| Authorization server | RFC 8414 metadata exposes authorization, token, and registration endpoints |
Overall status is `ready-for-oauth`, `ready-for-api-key` (with
`CTXPIPE_API_KEY` set), `warning`, or `failed`.
If TLS fails from a Node CLI against a local CA (Portless or a private CA), set
`NODE_EXTRA_CA_CERTS` to the CA file. Do not disable TLS verification.
## Authenticated checks with MCPJam
MCPJam is an external diagnostic client. It is not bundled into `ctxpipe` or the
self-hosted backend. Pin the versions below. Do not use `@latest`. Use a test
organization. Do not use a customer organization.
```bash
npx @mcpjam/cli@3.24.0 oauth login \
--url "$AUTH_BASE_URL/mcp" \
--protocol-version 2025-11-25 \
--registration dcr \
--credentials-out .mcpjam/credentials.json \
--debug-out .mcpjam/oauth-debug.json
npx @mcpjam/cli@3.24.0 tools list \
--url "$AUTH_BASE_URL/mcp" \
--credentials-file .mcpjam/credentials.json
npx @mcpjam/cli@3.24.0 tools call \
--url "$AUTH_BASE_URL/mcp" \
--credentials-file .mcpjam/credentials.json \
--tool-name list_workspaces \
--tool-args '{}'
```
For a visual OAuth and JSON-RPC timeline on a local origin:
```bash
npx @mcpjam/inspector@2.40.0 \
--url "$AUTH_BASE_URL/mcp" \
--name "ctxpipe self-host" \
--oauth \
--tab servers
```
The hosted MCPJam application only reaches public HTTPS. It cannot use
`localhost` or refresh against a local authorization server.
## Artifacts and credentials
Keep OAuth credentials and debug traces out of Git. Write them under `.mcpjam/`
(gitignored in the ctxpipe repository). Review files before sharing. The
credentials file can include an access token, refresh token, client id, and
client secret even when stdout is redacted.
The ctxpipe doctor never accepts or prints OAuth bearer tokens, and never
prints an API key. Do not put `access_token`
on the MCP URL.
MCPJam does not prove token refresh races, dropped-session recovery, or
concurrent clients. Do not gate CI on interactive OAuth or LLM evals.
## Common failures
* **`org_not_found` or `org_required`** — the tool call named an organization
that is not in the connection, an organization that the user is not a member
of, or no organization. Pass the right slug in `org`. For an OAuth connection,
add the organization on **Connected apps**, or reconnect the client and select
the organization during authorization.
* **`AUTH_BASE_URL` mismatch** — issuer, callbacks, and MCP discovery must use
the same public origin clients see.
* **Remote HTTP** — production MCP must be HTTPS.
* **Status unreachable or timed out** — backend, DNS, TLS, or an untrusted CA.
* **HTTP 401 without `resource_metadata`** — clients cannot start OAuth discovery.
* **Protected-resource audience mismatch** — RFC 9728 `resource` canonicalizes to
`/mcp` without the query string.
* **Authorization-server metadata missing registration** — dynamic client
registration cannot proceed.
* **CLI setup sign-in worked, but the MCP client still asks for authorization** —
expected. Setup auth and MCP OAuth are separate.
## Related pages
# Model configuration (https://docs.ctxpipe.ai/docs/self-hosting/models)
When you self-host ctxpipe, you set the LLM and embedding models with environment variables. The service supports any OpenAI-compatible HTTP API. Use **`MODEL_PROVIDER`** to select OpenRouter-specific features, Azure (`api-key` auth), or Bedrock (native SDK with IAM SigV4).
## `MODEL_PROVIDER` values
| Value | Auth (chat + embeddings) | `MODEL_PROVIDER_URL` default | Notes |
| ------------- | ------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `openai-like` | `Authorization: Bearer` | `https://openrouter.ai/api/v1` | Generic compatible API; no OpenRouter-only extras. |
| `openrouter` | `Authorization: Bearer` | same | Enables plugins, prompt cache, reasoning flags, medium-tier model fallbacks. |
| `azure` | `api-key` header only | **required** | Set your Azure OpenAI OpenAI-compatible base URL; override model name env vars for deployment IDs. |
| `bedrock` | IAM SigV4 (ECS task role) | not required | Native Bedrock Runtime via AWS SDK. Set `MODEL_BEDROCK_AWS_REGION` or rely on `AWS_REGION` / `AWS_DEFAULT_REGION`. No `MODEL_PROVIDER_API_KEY` on AWS CDK deployments. |
If you do not set **`MODEL_PROVIDER`**, it defaults to **`openai-like`**. To keep the **previous** behavior (OpenRouter URL heuristics that added extras whenever the host was OpenRouter), set **`MODEL_PROVIDER=openrouter`**.
For Vertex and other gateways that expect Bearer and a base URL, use **`openai-like`**. If you route through OpenRouter, use **`openrouter`**.
When you deploy with [`@ctxpipe/aws-cdk`](/docs/self-hosting/deployment/aws) and `modelProvider.kind: "bedrock"`, the construct does **not** store a static `MODEL_PROVIDER_API_KEY`. Backend and worker tasks receive IAM permissions (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`) and call Bedrock Runtime through the AWS SDK. They use the ECS task role (SigV4). Enable your chosen model IDs in the Bedrock console for the stack region before you use them. If the account is in an AWS Organization, also confirm that **service control policies (SCPs)** allow those same foundation-model ARNs. An SCP explicit deny overrides the task role. It shows as `AccessDeniedException` … `explicit deny in a service control policy`.
When `MODEL_PROVIDER=bedrock`, embeddings use the native Bedrock Runtime API. Only **Cohere** embed models work. Other Bedrock embedding models do not work yet. On AWS CDK, the default is `cohere.embed-v4:0` when you do not set `models.embedding`. You can also use an inference-profile ID, for example `global.cohere.embed-v4:0`. If an SCP denies a profile ID, the error often names the base `…/foundation-model/cohere.embed-v4:0` ARN.
## Quick Start (OpenRouter)
The default setup uses OpenRouter. Set your API key and start the stack:
```bash
MODEL_PROVIDER_API_KEY=sk-or-v1-... docker compose --profile deploy up -d
```
LLM and embeddings both use OpenRouter by default. You do not need extra configuration.
## Environment Variables
| Variable | Required | Default | Description |
| -------------------------- | -------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MODEL_PROVIDER` | No | `openai-like` | `openai-like` \| `openrouter` \| `azure` \| `bedrock` - see table above. |
| `MODEL_PROVIDER_API_KEY` | Yes\* | - | API key for OpenAI-compatible providers. Not used for native Bedrock (`MODEL_PROVIDER=bedrock`). |
| `MODEL_PROVIDER_URL` | No\*\* | OpenRouter base when not `azure` | Base URL for OpenAI-compatible LLM/embeddings HTTP. **Required** for `azure`. Not required for native Bedrock. |
| `MODEL_BEDROCK_AWS_REGION` | No | `AWS_REGION` / stack region | AWS region for Bedrock Runtime when `MODEL_PROVIDER=bedrock`. |
| `MODEL_PROVIDER_ALLOWLIST` | No | - | OpenRouter provider slugs, separated by commas (for example `openai,anthropic,azure`). When set, each OpenRouter call goes only to these providers. When unset, OpenRouter can use any provider. It needs `MODEL_PROVIDER=openrouter` or a `MODEL_PROVIDER_URL` on `openrouter.ai`. A value that is not a slug list, or other model settings, stop the backend at start. |
| `MODEL_FAST_NAME` | No | `openai/gpt-6-luna?reasoning.effort=high` | LLM model for the fast tier (OpenRouter slash ids). |
| `MODEL_MEDIUM_NAME` | No | `openai/gpt-6-luna?reasoning.effort=xhigh` | LLM model for the medium tier. |
| `MODEL_HIGH_NAME` | No | `openai/gpt-6-luna?reasoning.effort=xhigh` | LLM model for the high tier. |
| `MODEL_EMBEDDING_NAME` | No | `openai/text-embedding-3-large` | Embedding model ID. |
| `MODEL_VISION_NAME` | No | `openai/gpt-6.1-terra` | The one OpenRouter model that reads the images of the knowledge plane. It must take image input and have a ZDR endpoint on OpenRouter. A value of only spaces stops the backend at start. |
\*Required for each provider except `bedrock`. The backend refuses model calls without it. For a provider that needs no key, such as Ollama, set any value that is not empty.
\*\*Default `MODEL_PROVIDER_URL` is `https://openrouter.ai/api/v1` for `openai-like` and `openrouter` so bundled default model IDs stay valid.
Each model call to OpenRouter allows only zero data retention (ZDR) endpoints, with no data collection. A model with no ZDR endpoint fails. A call goes to OpenRouter when `MODEL_PROVIDER` is `openrouter`, or when `MODEL_PROVIDER_URL` is on `openrouter.ai`. Another OpenAI-compatible API does not get these routing fields.
## How to Pick Models
### LLM tiers
The service uses three tiers for different workloads:
| Tier | Use case | Default model spec |
| ---------- | --------------------------------- | ------------------------------------------ |
| **fast** | Quick tasks, naming, planning | `openai/gpt-6-luna?reasoning.effort=high` |
| **medium** | Main agent, balanced cost/quality | `openai/gpt-6-luna?reasoning.effort=xhigh` |
| **high** | Complex reasoning, best quality | `openai/gpt-6-luna?reasoning.effort=xhigh` |
Override any tier with `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, or `MODEL_HIGH_NAME`. Use model IDs from your provider (OpenRouter: `org/model-name`; Bedrock on AWS CDK: `vendor.model` with dot, e.g. `openai.gpt-6-luna?reasoning.effort=high`).
### Model spec query params
Append **`?reasoning.effort=value`** to set the reasoning depth of a tier without extra environment variables. The supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Each provider adapter maps this value to its upstream API (`reasoning_effort` on OpenAI-like / Azure / Bedrock Chat Completions; nested `reasoning.effort` on OpenRouter).
```text
openai/gpt-6-luna?reasoning.effort=high
openai.gpt-6-luna?reasoning.effort=xhigh
```
To disable reasoning in environment specs, use **`reasoning.effort=none`**. Do not use `reasoning=false`. Call sites can also pass `getModel(..., { reasoning: false })`, which merges `reasoning.effort=none` over the tier spec. ctxpipe sends model ids upstream without query strings.
When **`MODEL_PROVIDER=openrouter`** (or hosted ctxpipe, which sets it), the **medium** tier sends the OpenRouter `models` array. If the primary model fails (rate limits, downtime, moderation), routing tries the next distinct model IDs in order (**`MODEL_FAST_NAME`**, then **`MODEL_HIGH_NAME`**). ctxpipe removes duplicate base model ids from that chain. This does **not** apply to `openai-like`.
### Embedding model
**Important:** The embedding model must support **2000 dimensions**. The schema stores vectors in `vector(2000)`; a model that is not compatible fails.
* **OpenAI-compatible providers** (OpenRouter, OpenAI, Vertex): Use models that accept a `dimensions` parameter, for example `openai/text-embedding-3-large` (3072 dims, truncated to 2000).
* **Amazon Bedrock**: Use a **Cohere** embed model ID (for example `cohere.embed-v4:0` or `global.cohere.embed-v4:0`). Other Bedrock embed families are not supported yet.
* **Ollama**: Use models with native 2000-dim support, e.g. `qwen3-embedding`. Set `MODEL_PROVIDER_URL=http://host:11434/v1` and `MODEL_EMBEDDING_NAME=qwen3-embedding` (embeddings use `{MODEL_PROVIDER_URL}/embeddings`).
All providers use the same OpenAI-compatible embeddings API. Ollama exposes it at `/v1/embeddings`.
## Provider Examples
### OpenRouter (default host)
```bash
MODEL_PROVIDER_API_KEY=sk-or-v1-...
# Optional explicit mode (enables OpenRouter extras; default URL is OpenRouter either way):
# MODEL_PROVIDER=openrouter
MODEL_PROVIDER_URL=https://openrouter.ai/api/v1
```
### Azure OpenAI
```bash
MODEL_PROVIDER=azure
MODEL_PROVIDER_API_KEY=
MODEL_PROVIDER_URL=https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT
# Set model env vars to your deployment names as needed.
```
### Amazon Bedrock (native SDK)
```bash
MODEL_PROVIDER=bedrock
MODEL_BEDROCK_AWS_REGION=us-east-1
MODEL_EMBEDDING_NAME=cohere.embed-v4:0 # or global.cohere.embed-v4:0; Cohere only on Bedrock
# ECS task role / instance profile supplies AWS credentials; no MODEL_PROVIDER_URL or API key.
# Org SCPs must allow the underlying foundation-model ARN (see AWS CDK Bedrock callout above).
```
### Ollama (local chat + embeddings)
```bash
MODEL_PROVIDER_URL=http://localhost:11434/v1
MODEL_PROVIDER_API_KEY=ollama # Ollama needs no key, but the value must not be empty
MODEL_FAST_NAME=
MODEL_MEDIUM_NAME=
MODEL_HIGH_NAME=
MODEL_EMBEDDING_NAME=qwen3-embedding
# For Docker: use http://host.docker.internal:11434/v1
```
Set each model name. The defaults are OpenRouter model ids, which Ollama does
not know.
## Resources
* [OpenRouter models](https://openrouter.ai/models)
* [OpenRouter embeddings](https://openrouter.ai/docs/api/reference/embeddings)
* [Ollama OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility#%2Fv1%2Fembeddings)
# Architecture (https://docs.ctxpipe.ai/docs/self-hosting/architecture)
Self-hosted ctxpipe uses the same product architecture as hosted ctxpipe. But the
operator owns the runtime, network boundary, secrets, and stateful services.
Keep the browser, REST API, auth callbacks, and MCP endpoint on the same public
backend origin.
## Request flow
1. The browser opens your public `AUTH_BASE_URL`.
2. The backend serves API, auth, webhook, and MCP routes from that origin.
3. The backend proxies UI requests to the `ui` service.
4. Auth sessions, repository records, connections, and workflow state live in
Postgres.
5. Repository ingestion calls `codesearch`, which clones repositories and
maintains Zoekt indexes. Codesearch is a **single replica**. Zoekt shards,
git clone cache, and SCIP artifacts live on one process and one volume.
Extra worker replicas do not raise ingest throughput unless codesearch
memory (and `OPENWORKFLOW_CONCURRENCY` / indexer caps) also grow.
6. The worker runs durable jobs for ingestion, connector setup, scoped source
mirrors, Slack thread capture, and Forge provisioning.
7. Extracted claims and relationships are projected into the configured graph
database. For an organization on the new knowledge store, the worker also
writes the documents, their sections, and their relations to ClickHouse.
8. Chat and MCP tools retrieve from Postgres, codesearch, and the graph database
(or ClickHouse, for an organization on the new knowledge store) before
calling your configured model provider.
## Services
| Service | Default port | Stateful | Purpose |
| ----------------------------- | ---------------------------------------------------: | -------- | ------------------------------------------------------------ |
| `backend` | `3000` | No | Public API, auth, MCP, callbacks, UI proxy |
| `ui` | `3002` | No | Product web app, proxied by backend |
| `worker` | internal | No | Background workflows and connector jobs |
| `codesearch` | `3001` | Yes | Repository clone cache and Zoekt index. Stay at one replica. |
| `postgres` | `5432` internal (no host port in the deploy profile) | Yes | Primary relational state |
| `falkordb` | `6379` Redis, `7687` Bolt, internal | Yes | Default graph database |
| `clickhouse` (Docker Compose) | `8123` HTTP, internal | Yes | The new knowledge store |
| `dind` (Docker Compose) | `2376` internal | No | Docker daemon for Workspace chat sandboxes |
### Chat sandbox network policy
The same policy holds on Docker Compose and AWS:
* A chat sandbox holds no credential. It has no GitHub token, no model proxy
credential, and no tool bridge token in its environment, files, or Git
configuration. [Agent Vault](https://github.com/Infisical/agent-vault)
(open source) runs next to the sandboxes and adds each credential to the
requests that need it.
* Sandboxes reach the network only through the Agent Vault proxy. On Compose,
rules in the `dind` service enforce this. On AWS, `DOCKER-USER` rules on the
sandbox host enforce it. HTTP and HTTPS stay open. Agents can clone, fetch,
push, call the GitHub CLI, install packages, and read documentation on the
internet. The proxy forwards hosts that have no credential rule without
change.
* The proxy blocks all other traffic from a sandbox. This includes a direct connection
that does not use the proxy, any non-HTTP protocol (SSH, raw TCP, UDP), and
DNS. Sandboxes do not resolve names. The proxy resolves them.
* The proxy allows private addresses as follows. On Compose, it allows only the
fixed address of the backend on the `sandbox` network. On AWS, it allows only
the subnets of the backend. The host also rejects traffic from Agent Vault to
the host itself. Agent Vault never reaches its own management API. Sandboxes
never reach data stores. Compose never publishes Postgres or FalkorDB. AWS
admits only the security group of the app to them.
* Sandboxes cannot reach cloud instance metadata (`169.254.169.254`).
* Each agent port needs a password, and sandboxes cannot reach each other over
the Docker bridge.
* Each chat turn gets its own Agent Vault vault and proxy session. The backend
deletes the vault when the turn ends, and the session stops with it. The
backend also revokes the turn's GitHub token.
* If Agent Vault does not answer, Docker chat fails closed with a clear error.
The backend never gives a sandbox a credential instead.
The deployment generates the Agent Vault passwords. You do not set them. On
Compose, the master password is on a volume that only Agent Vault mounts, and
the owner password is on a volume that only the backend and the worker mount.
On AWS, the stack keeps both in Secrets Manager. Only the host reads the
master password.
#### Agent Vault owner password
The backend registers the owner password on its first login to an Agent Vault
with no owner. After that, Agent Vault refuses another password. Docker
chat then fails closed with an error that names this section. To change the
password, first change it in Agent Vault (`agent-vault account change-password`,
or its web UI on port 14321 inside the deployment). Then give the backend the
new value. To start over, remove Agent Vault's data (Compose: the
`agent_vault_data` volume; AWS: replace the sandbox host). Run vaults are
temporary, so no data is lost.
Tools in a sandbox must use the standard proxy variables (`HTTPS_PROXY`,
`HTTP_PROXY`) and trust the proxy's certificate authority. The sandbox
environment sets both. A tool that ignores the proxy variables cannot reach
the network.
### Chat sandboxes on Docker Compose
Compose runs each Workspace conversation's agent in its own container inside
`dind`, a stock Docker-in-Docker service. `dind`, `agent-vault`, `backend`, and
`worker` share a separate `sandbox` network (`172.30.99.0/24`):
| Component | Port | Reached by | Purpose |
| ----------------------------------------------- | -------------------------------------------------------------: | --------------------------------------- | ------------------------------------------------- |
| Docker API (mutual TLS) | `2376` | `backend`, `worker` | Create, stop, resume, and remove sandboxes |
| Sandbox agent ports (password per conversation) | `32768`–`60999` on `dind` | `backend` | The backend talks to the agent in each sandbox |
| Agent Vault API | `14321` | `backend`, `worker` | One vault per chat turn |
| Agent Vault proxy | `14322` | Sandboxes (the only address they reach) | Adds credentials and forwards HTTP(S) |
| Backend model proxy and tool bridges | `3000` and per-run ports on `backend.sandbox.ctxpipe.internal` | Agent Vault, for sandboxes | Model calls and tool calls go back to the backend |
See [Docker: Chat sandboxes](/docs/self-hosting/deployment/docker#chat-sandboxes).
### Chat sandbox host on AWS
On AWS, `@ctxpipe/aws-cdk` adds one EC2 Docker host next to the Fargate
services. Each Workspace conversation's agent runs in its own container there.
| Component | Port | Reached by | Purpose |
| ----------------------------------------------- | ----------------------: | --------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Docker API (mutual TLS) | `2376` | `backend`, `worker` | Create, stop, resume, and remove sandboxes |
| Sandbox agent ports (password per conversation) | `32768`–`60999` | `backend` | The backend talks to the agent in each sandbox |
| Agent Vault API | `14321` | `backend`, `worker` | One vault per chat turn |
| Agent Vault proxy | `14322` (on the host) | Sandboxes (the only address they reach) | Adds credentials and forwards HTTP(S) |
| Backend model proxy and tool bridges | `3000`, `32768`–`60999` | Agent Vault, for sandboxes | Model calls and tool calls go back to the backend task that runs the turn, at its VPC DNS name |
`DOCKER-USER` rules on the host let sandboxes reach only the Agent Vault proxy.
Agent Vault reaches the internet through NAT and may dial private addresses
only in the VPC. Security groups keep Postgres, Neptune, ClickHouse, EFS, codesearch, the
UI, and the worker away from the host. The host also blocks sandboxes from the
host itself. See
[AWS: Chat sandbox host](/docs/self-hosting/deployment/aws#chat-sandbox-host).
## Public surface
Expose the backend origin publicly. Keep internal services private unless you
have an operational need.
| Route or endpoint | Public? | Notes |
| ------------------------------------ | -------- | ---------------------------------------------------------------------------------------------- |
| `/` | Yes | Backend proxies UI requests |
| `/.auth/*` | Yes | Better Auth routes and provider callbacks |
| `/api/v1/*` | Yes | REST API and connector callbacks |
| `/:orgSlug/api/v1/*` | Yes | Organization-scoped REST API |
| `/mcp` | Yes | Streamable HTTP MCP endpoint. Tool calls name the organization in `org` |
| `/.status` | Optional | Basic backend status endpoint |
| `ui:3002` | No | Internal service behind backend. Docker Compose binds it to `127.0.0.1` only. |
| `codesearch:3001` | No | Internal service called by backend and workflows. Docker Compose binds it to `127.0.0.1` only. |
| `postgres`, `falkordb`, `clickhouse` | No | Stateful infrastructure. Never published. |
## Stateful data
Back up anything you cannot rebuild quickly.
| Data | Location | Recovery note |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| App database | Postgres. On Compose, an anonymous volume unless you mount a named one (see [Docker](/docs/self-hosting/deployment/docker#persistent-state)). | Required. Contains users, sessions, repositories, connections, workflow state, and encrypted connector metadata. |
| Graph data | FalkorDB or another graph database. On Compose, the writable layer of the `falkordb` container unless you mount a named volume (see [Docker](/docs/self-hosting/deployment/docker#persistent-state)). | Back up for fast recovery. Some graph data may be regenerated by reindexing, but that can be slow and may not cover all operational state. |
| Knowledge content | ClickHouse. On Compose, the `clickhouse_data` volume. | Back up for fast recovery. ClickHouse holds only data that ctxpipe makes from the Workspace repositories, so a full rebuild of each Workspace can make it again. |
| Repository cache | `repo_cache` volume in Compose, EFS `/data` on AWS CDK | Rebuildable by recloning, but useful to retain. |
| Zoekt index | `zoekt_index_deploy` volume in Compose, EFS `/data` on AWS CDK | Rebuildable by reindexing, but can be expensive for large repositories. |
| Chat sandboxes | `sandbox_docker` volume in Compose, Docker volume on the AWS sandbox host | No backup needed. Conversations keep their work in their git branch, and a lost sandbox is created again from it. |
| Secrets | Deployment secret store or `.env` files | Required to decrypt sessions and connector credentials. Protect and version changes carefully. |
## Related pages
# Quickstart (https://docs.ctxpipe.ai/docs/self-hosting/quickstart)
This quickstart is the same for each deployment path. Deploy ctxpipe. Then test
the product loop: sign in, connect GitHub, and index one repository. Then search
it from an agent with the MCP tools.
## 1. Deploy
For a Docker Compose evaluation, clone the repository and set the minimum
values:
```bash
git clone https://github.com/ctxpipe-ai/ctxpipe.git
cd ctxpipe
cp docker-compose.env.example .env
# Set AUTH_SECRET, AUTH_BASE_URL, CTXPIPE_PUBLIC_APP_URL,
# MODEL_PROVIDER, and MODEL_PROVIDER_API_KEY.
pnpm start
```
`pnpm start` runs the Compose `deploy` profile. See
[Docker](/docs/self-hosting/deployment/docker) for what it starts and what to
set before people depend on it.
For AWS, see [AWS CDK](/docs/self-hosting/deployment/aws).
The browser, auth, the API, connector callbacks, and MCP all use the public
backend origin, `AUTH_BASE_URL`. Keep the other services private.
## 2. Check the deployment
Before you add a source, make sure that:
* `GET {AUTH_BASE_URL}/.status` returns `ok`;
* the app opens through `AUTH_BASE_URL`;
* you can sign in with at least one method. See
[Authentication](/docs/self-hosting/authentication);
* the backend and the worker can reach the model provider;
* Postgres and the graph database are healthy.
## 3. Create your GitHub App and a Workspace
Open **Connectors**, select **Add connection**, then select **Connect** on the
**GitHub** row. The wizard helps you create a GitHub App that your deployment
owns. Follow [Self-host GitHub](/docs/self-hosting/github). Then install the
App on a few repositories.
Then [create a Workspace](/docs/workspaces/create-workspace), and
[link one code repository](/docs/workspaces/linked-repositories) in its
**Settings**. Wait until the repository shows **Indexed**.
## 4. Connect an agent
Run the CLI with the URL of your deployment:
```bash
npx ctxpipe init --base-url https://ctx.example.com
```
Or set up your client by hand with `https://ctx.example.com/mcp`. See
[Connect your agent](/docs/mcp/mcp-docs). The client authorizes against your
deployment. The signed-in user must be a member of each organization that the
client reads.
## 5. Prove the loop
Ask the agent:
```text
Use ctxpipe before you plan. Call list_workspaces, then search_knowledge and
search_code. Name the relevant architecture, standards, and source files, and
say what still needs a check in the code.
```
You are ready to invite your team when:
* `list_workspaces` lists your organization and its Workspaces;
* `search_knowledge` and `search_code` return useful refs;
* a repository that you index again finishes;
* backups and logs are in place;
* you have a rollback procedure for your deployment path.
# Operations (https://docs.ctxpipe.ai/docs/self-hosting/operations)
These docs assume that you already configured the deployment. Use this page to
keep a running instance healthy and to diagnose component-level failures.
## Health checks
| Check | Command or route | Expected result |
| ------------------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| Backend status | `GET /.status` | JSON response with `status: "ok"` and `timestamp` |
| Postgres in Compose | Docker health check uses `pg_isready` | Service is healthy before migrations and app services start |
| UI through backend | Open `AUTH_BASE_URL` | App loads from the backend origin |
| MCP route | `npx ctxpipe doctor mcp --url "$AUTH_BASE_URL/mcp"` then an OAuth-capable MCP client | Doctor reports `ready-for-oauth` (or `ready-for-api-key` when `CTXPIPE_API_KEY` is set); client OAuth then succeeds |
| Codesearch | Add or reindex a repository | Repository reaches indexed state; backend can call `CODESEARCH_URL` |
| Graph database | Open graph page or use graph-backed chat/MCP | Graph responses succeed; backend does not return graph connectivity errors |
The reference codesearch service does not expose a separate public health
endpoint. Use successful repository clone, index, and search operations as the
health signal.
## Migrations
The Compose `deploy` profile starts a `migrate` service before backend, worker,
UI, and codesearch. Migrate uses the owner URL and provisions `ctxpipe_app`.
App processes use the app-role `DATABASE_URL`. The migrate container runs:
```bash
bun run apps/backend/src/db/migrate.ts
```
The AWS CDK migrate task and the hosted deploy run the same script,
`migrate.ts`. It does these steps in this order. It makes the `ctxpipe_app`
role ready. It applies the Drizzle and OpenWorkflow migrations. It runs every
data backfill. Then it grants `ctxpipe_app` access to the tables.
After the Drizzle migrations, migrate runs the backfills. The org backfill
fills a missing `org_id` on rows that the previous release wrote (the
`BEFORE INSERT` triggers on `repository_checkouts`, `confluence_spaces`, and
`claim_evidence` do the same at write time). You do not run it by hand. See
[Upgrades](/docs/self-hosting/upgrades#what-the-sql-changes) for the
expand-only rules and the rollback runbook.
Migrations are idempotent. Drizzle tracks applied migrations in the
`drizzle.__drizzle_migrations` table and skips migrations that already ran. A second run
changes nothing.
Migrate reads the `ctxpipe_app` password from `DATABASE_APP_PASSWORD`:
* When the password is set, migrate creates the role if it is missing and sets
its password.
* When the password is not set and the role exists, migrate only grants access.
* When the password is not set and the role does not exist, migrate stops with
exit code 1 before it changes the database. The error tells you how to fix it
for each deployment path. Compose sends `POSTGRES_PASSWORD`. The CDK
migrate task sends the app database credentials secret.
For non-Compose deployments:
1. Back up Postgres first.
2. Run migrations once against the production database.
3. Start the new backend, worker, UI, and codesearch versions after migrations
complete.
## Logs
ctxpipe writes structured application logs through evlog. Without an OTLP logs
endpoint, logs go to stdout.
| Component | Useful signals |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `backend` | auth failures, REST errors, MCP requests, graph read failures, connector callbacks |
| `worker` | ingestion jobs, Confluence, Linear, PagerDuty, Buildkite, and Plain sync, Forge provisioning, background workflow failures |
| `codesearch` | clone failures, indexing failures, Zoekt errors |
| `postgres` | connection limits, slow queries, disk pressure |
| `graph database` | connection failures, memory pressure, query errors |
| `clickhouse` | disk use of the hot volume, the count of pending mutations in `system.mutations`, query errors |
Set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`,
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, and `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`
on the app services when you want telemetry to leave stdout and reach your
observability backend. See
[Configuration](/docs/self-hosting/configuration) for headers, service names,
and LLM spans. ctxpipe does not run a collector for you. You see export failures in your
collector or vendor. There is no bundled collector container.
## Backups
Back up before upgrades, secret rotation, connector migrations, and storage
changes.
| Data | Backup guidance |
| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Postgres | Required. Use managed database backups or `pg_dump`/physical backups. Test restore regularly. |
| Graph database | Back up or snapshot according to provider guidance. On Compose, mount a named volume first (see [Docker](/docs/self-hosting/deployment/docker#persistent-state)). |
| ClickHouse | Optional. Back up `clickhouse_data` for fast recovery. Without a backup, a full rebuild makes the content again from the Workspace repositories (see [Recover the knowledge store](/docs/self-hosting/configuration#recover-the-knowledge-store)). |
| `repo_cache` | Optional but recommended for large/private repositories. Rebuildable by recloning. |
| `zoekt_index_deploy` | Optional but recommended for large deployments. Rebuildable by reindexing. |
| Secrets | Export from your secret manager or platform. Keep historical values needed for rollback. |
## Chat sandbox host (AWS)
`@ctxpipe/aws-cdk` runs Workspace chat sandboxes on one EC2 Docker host (see
[AWS: Chat sandbox host](/docs/self-hosting/deployment/aws#chat-sandbox-host)).
It holds no data that you need to back up. A replaced host starts empty, and
conversations continue from their git branch.
Find the instance from the `SandboxHostAutoScalingGroupName` stack output and
open a shell with Session Manager (no SSH key or open port is needed):
```bash
aws ssm start-session --target
```
| Check | Command on the host | Expected result |
| ------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------ |
| Docker is up | `sudo systemctl is-active docker` | `active` |
| Docker volume | `df -h /var/lib/docker` | Below 80% used. The disk alarm fires at 80%. |
| What uses the disk | `sudo docker system df` | Images, containers, and build cache sizes |
| Sandboxes | `sudo docker ps -a` | One container per recent conversation; stopped ones are idle sandboxes |
| Memory cap | `systemctl status ctxpipe-sandboxes.slice` | Memory in use stays below the slice limit (85% of host memory) |
| Chat image | `sudo docker image ls ghcr.io/ctxpipe-ai/chat-sandbox` | The tag the backend's `SANDBOX_CHAT_IMAGE` names, after the first chat of a release |
| Workspace bases | `sudo docker image ls ctxpipe-workspace-base` | At most about one image for each Workspace that had a new conversation in the last 30 days |
| Boot log | `sudo tail -n 100 /var/log/cloud-init-output.log` | Ends with the CloudFormation success signal |
Sandboxes stop after 5 minutes idle. ctxpipe removes them 30 days after last use.
An organization runs at most 50 sandboxes at once. A Workspace base is an image
of the Workspace repository after clone and setup. A new conversation then
starts without a clone. The periodic sandbox cleanup deletes a base when a newer
base replaces it. It deletes the current base when no new conversation started
from it for 30 days. The same cleanup removes objects with ctxpipe labels
(`ai.ctxpipe.*`) that no database row records. If the volume still fills, check
`docker system df` to see what grows. Then raise `sandboxHost.dockerVolumeSizeGiB`
and run `cdk deploy`. The new size replaces the host.
Do not run `docker system prune` or `docker container prune` while
conversations are active. These commands delete stopped sandboxes that
conversations would resume. Do not run `docker image prune -a`. It deletes
Workspace bases. The next new conversation of each Workspace then clones the
repository again while a new base builds.
## Chat sandboxes (Docker Compose)
The Compose deploy profile runs Workspace chat sandboxes inside the `dind`
service (see [Docker: Chat sandboxes](/docs/self-hosting/deployment/docker#chat-sandboxes)).
Its `sandbox_docker` volume holds no data that you need to back up. Run these commands from
the repository root:
| Check | Command | Expected result |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Daemon is up | `docker compose --profile deploy ps dind` | `healthy` |
| Chat image | `docker compose --profile deploy logs chat-sandbox-image` | Ends with the image ID (`sha256:…`) |
| Sandbox disk | `docker compose exec dind df -h /var/lib/docker` | Room left on the host disk that holds the Docker volumes |
| What uses the disk | `docker compose exec dind docker system df` | Images, containers, and build cache sizes |
| Workspace bases | `docker compose exec dind docker image ls ctxpipe-workspace-base` | At most about one image for each Workspace that had a new conversation in the last 30 days |
| Sandboxes | `docker compose exec dind docker ps -a` | One container per recent conversation; stopped ones are idle sandboxes |
| Memory cap | `docker compose exec dind cat /sys/fs/cgroup/ctxpipe-sandboxes/memory.current /sys/fs/cgroup/ctxpipe-sandboxes/memory.max` | Memory in use stays below the cap. Missing on hosts without cgroup v2 (`dind` logs a warning). |
| Metadata block | `docker compose exec dind iptables -t raw -S PREROUTING` | Includes `-d 169.254.169.254/32 -j DROP` |
| No published data stores | `docker port $(docker compose ps -q postgres falkordb clickhouse)` | Prints nothing |
| No host socket | `docker inspect --format '{{range .Mounts}}{{.Source}} {{end}}' $(docker compose ps -q backend worker)` | No `/var/run/docker.sock` |
ctxpipe removes sandboxes and Workspace bases on the same schedule as on AWS.
To free disk and keep the sandboxes, remove untagged chat images and build
cache:
```bash
docker compose exec dind docker image prune -f
docker compose exec dind docker builder prune -f
```
`docker image prune` without `-a` keeps every tagged image, so it keeps the
Workspace bases. Do not run `docker system prune` or `docker container prune`
inside `dind` while conversations are active. These commands delete stopped
sandboxes that conversations would resume. Do not run `docker image prune -a`.
It deletes Workspace bases. The next new conversation of each Workspace then
clones the repository again while a new base builds.
## Recovery runbooks
### Workspace chat fails to start on Docker Compose
* A 503 that says the Docker daemon is not reachable means that `dind` is down.
Confirm that it is healthy (`docker compose --profile deploy ps -a`). Check
`docker compose logs dind`.
* A 503 that says the chat sandbox image is missing means that
`chat-sandbox-image` has not finished or has failed. For example, a package
mirror was unreachable, or someone removed the `sandbox_docker` volume. Check its
logs. Then run it again:
`docker compose --profile deploy up -d chat-sandbox-image`.
* Check backend logs for Docker TLS errors. After the `sandbox_ca` volume is
removed, restart `backend` and `worker` so they read the new certificates.
* `dind` needs a privileged container. On a host that refuses one, chat is not
available. Use the AWS CDK path, or run the sandboxes on another Docker host
(see [Chat sandboxes](/docs/self-hosting/configuration#chat-sandboxes)).
### Workspace chat fails to start on AWS
* Confirm the sandbox host instance is `InService` in its Auto Scaling group.
* Check backend logs for Docker connection or TLS errors. Backend and worker
reach the host at `sandbox-host.ctxpipe.local:2376`. The host registers that
name when it finishes booting.
* If the backend reports that the chat sandbox image is missing, the host
could not pull `ghcr.io/ctxpipe-ai/chat-sandbox`. Check the host's outbound
access to `ghcr.io` through NAT.
* Terminate the instance to get a fresh host. The Auto Scaling group replaces
it in a few minutes and reuses the stored TLS certificates.
### App loads but sign-in fails
* Confirm `AUTH_BASE_URL` matches the browser origin.
* Confirm `AUTH_SECRET` is present and stable.
* Confirm provider callback URLs use the public backend origin.
* Check backend logs for Better Auth or OAuth provider errors.
### Repository never becomes indexed
* Check `codesearch` logs for clone/index errors.
* Confirm GitHub App permissions and installation scope.
* Confirm `CODESEARCH_URL` points at the codesearch service from the backend.
* Reindex after fixing credentials or network access.
### MCP discovery or OAuth fails
* Confirm `GET /.status` returns `{ "status": "ok" }`.
* Run `npx ctxpipe doctor mcp --url "$AUTH_BASE_URL/mcp"`.
* A `ready-for-oauth` result means routing, the Bearer challenge, and OAuth
discovery are consistent. It does not prove authenticated `tools/list` or
a tool call.
* Confirm `AUTH_BASE_URL` matches the public origin clients use, including TLS.
* Confirm the MCP URL path is `/mcp`.
* For authenticated OAuth, `tools/list`, and a `list_workspaces` call, use a version-pinned
MCPJam client against a test organization. See
[MCP](/docs/self-hosting/mcp).
### Chat or MCP returns weak context
* Confirm repositories are indexed.
* Confirm graph extraction has completed for the repository.
* Check model provider credentials and rate limits.
* Check backend and worker logs around code ingestion and graph projection.
### Knowledge graph fails to load
* Confirm `GRAPH_DB_URI` and provider credentials.
* Confirm the graph database is reachable from backend and worker.
* Check backend logs around graph snapshot/read operations.
* Re-run repository ingestion if graph data needs to be rebuilt.
### An index run of the knowledge store takes longer than usual
For an organization on the new knowledge store, some events start a full
rebuild of a Workspace. An example is a force-push that removed the last
indexed commit. A full rebuild reads and embeds each file
again, so it takes longer. Chat and MCP continue to answer from the index. See
[Changes and full rebuilds](/docs/workspaces/indexing#changes-and-full-rebuilds).
The worker log of each run has `knowledgeProjection.mode` (`delta` or
`full`) and `knowledgeProjection.fullReason`. The span of the run has the same
reason in `ctxpipe.knowledge.full_reason`: `first_run`, `missing_commit`,
`projection_version`, `sources_config`, or `attempts`. The linked section
gives the cause of each.
### Confluence setup stalls
* Confirm Atlassian OAuth callback URL.
* Confirm Forge install URL or provisioning token.
* Confirm the worker can run Forge provisioning and Confluence sync jobs.
* See [Confluence & Atlassian](/docs/self-hosting/confluence-atlassian).
### Linear setup stalls or content becomes stale
* Confirm that you registered a Linear OAuth application in the wizard, or that
`LINEAR_*` is set on the backend and the worker.
* Confirm the Linear webhook secret is saved on the connection or
`LINEAR_WEBHOOK_SECRET` is present on backend, and signed requests reach
`/api/v1/webhook/linear`.
* Confirm the selected GitHub App installation still has write access and that
merge pushes reach `/api/v1/webhook/github/`.
* Check worker logs for `linear-sync-config`, `linear-sync-content`, or
`linear-sync-incremental` failures.
* Choose **Sync now** in the menu of the connector card after you fix the
underlying credential, network, or permission failure.
* See [Linear](/docs/self-hosting/linear).
### PagerDuty setup stalls or incidents become stale
* Confirm PagerDuty OAuth credentials are present on backend and worker.
* Confirm the connection is live and the incident's service is listed in the
scope of a PagerDuty source with the status **Connected**. There is no shared webhook secret. ctxpipe
stores the secret of each subscription on the connection.
* Confirm signed requests reach `/api/v1/webhook/pagerduty` and merge pushes
reach `/api/v1/webhook/github/`.
* Check worker logs for `pagerduty-sync-config`, `pagerduty-sync-content`, or
`pagerduty-sync-entity` failures.
* Choose **Sync now** in the menu of the connector card after you fix the
underlying credential, network, or permission failure.
* See [PagerDuty](/docs/self-hosting/pagerduty).
## Related pages
# Production readiness (https://docs.ctxpipe.ai/docs/self-hosting/production-readiness)
Use this checklist before you make a self-hosted deployment available to a team.
The reference Compose stack is a first draft. A production deployment must
define the network boundary, persistence, secrets, and rollback process.
## Public origin
ctxpipe expects a single public backend origin for browser, REST, auth, and MCP
traffic.
| Check | Why it matters |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `AUTH_BASE_URL` is the public HTTPS origin | Better Auth callbacks, generated links, OAuth issuer values, MCP auth, and connector callbacks depend on this value. |
| `CTXPIPE_PUBLIC_APP_URL` matches the public backend origin | The UI uses this value at build time. |
| Browser users enter through the backend | The backend proxies the UI, keeping auth and API requests on the same origin. |
| `AUTH_ALLOWED_ORIGINS` includes only required extra origins | Avoid broad CORS allowances. |
## TLS and proxying
Put a TLS-terminating proxy or platform load balancer in front of the backend.
Expose only the backend origin publicly unless you have an operational
reason to expose other services.
| Service | Public exposure |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `backend` | Public HTTPS |
| `ui` | Private; backend proxy should serve it |
| `worker` | Private |
| `codesearch` | Private |
| `postgres` | Private |
| `falkordb` or graph database | Private |
| `clickhouse` | Private |
| `dind` (Docker Compose) or the AWS sandbox host | Private. Do not publish the Docker API or other unauthenticated services on a host that runs chat sandboxes. |
The Compose deploy profile binds `ui` (`3002`) and `codesearch` (`3001`) to
`127.0.0.1` only. Do not change them to all interfaces.
The backend refuses a webhook body above its limit with `413`. The limits are
25 MiB for GitHub, 5 MiB for Linear, Buildkite, and Plain, and 1 MiB for Slack,
Notion, PagerDuty, and Atlassian. If your proxy has a lower body limit, set it
to at least 25 MiB for `/api/v1/webhook/github`. Set it to 5 MiB for
`/api/v1/webhook/linear`, `/api/v1/webhook/buildkite/*`, and
`/api/v1/webhook/plain/*`.
Telemetry export is outbound only. There is no collector to expose. Set
`OTEL_EXPORTER_OTLP_*` to your own endpoint when you want traces, logs, or
metrics to leave the app containers.
## Callback URLs
Register callback and webhook URLs against your public origin.
| Integration | URL shape |
| ------------------------------------------ | ------------------------------------------------------------- |
| Atlassian OAuth | `{AUTH_BASE_URL}/api/v1/integrations/atlassian/callback` |
| Linear OAuth | `{AUTH_BASE_URL}/api/v1/integrations/linear/callback` |
| Linear webhook | `{AUTH_BASE_URL}/api/v1/webhook/linear` |
| GitHub App webhook | `{AUTH_BASE_URL}/api/v1/webhook/github/` |
| Slack OAuth | `{AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback` |
| Slack Events API | `{AUTH_BASE_URL}/api/v1/webhook/slack` |
| Notion OAuth | `{AUTH_BASE_URL}/api/v1/connectors/notion/oauth/callback` |
| Notion webhook | `{AUTH_BASE_URL}/api/v1/webhook/notion?provisioningToken=...` |
| PagerDuty OAuth | `{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback` |
| PagerDuty webhook | `{AUTH_BASE_URL}/api/v1/webhook/pagerduty` |
| Buildkite webhook | `{AUTH_BASE_URL}/api/v1/webhook/buildkite/` |
| Plain webhook (ctxpipe creates the target) | `{AUTH_BASE_URL}/api/v1/webhook/plain/` |
| MCP | `{AUTH_BASE_URL}/mcp` |
| App auth | `/.auth/*` routes under `AUTH_BASE_URL` |
## Persistence
Do not treat the reference Compose volumes as disposable in production. On
Compose, mount named volumes for Postgres and FalkorDB before the first start.
See [Docker: Persistent state](/docs/self-hosting/deployment/docker#persistent-state).
| State | Production expectation |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Postgres | Managed backups, restore test, monitoring, and storage alerts |
| Graph database | Backup or snapshot according to provider guidance |
| ClickHouse (`clickhouse_data` on Compose) | Backup for fast recovery, or a full rebuild from the Workspace repositories (see [Recover the knowledge store](/docs/self-hosting/configuration#recover-the-knowledge-store)). Disk alerts. |
| `repo_cache` | Persistent volume if recloning large repositories is expensive |
| `zoekt_index_deploy` | Persistent volume if rebuilding indexes is expensive |
| Secrets | Stored in a secret manager or platform secret store, not committed files |
## Codesearch capacity
Keep `codesearch` at **one replica**. Zoekt shards, git clone cache, and SCIP
artifacts are process-local. Extra worker replicas do not raise ingest
throughput unless codesearch memory and the injected
`OPENWORKFLOW_CONCURRENCY` / `CODESEARCH_INDEXER_CONCURRENCY` /
`CODESEARCH_INDEX_PIPELINE_CONCURRENCY` values also grow. Compose `deploy`
uses the small pair (indexer 1, pipelines 1, worker concurrency 6). AWS CDK
injects the matching pair for `small` / `medium` / `large`.
## Worker memory
The worker starts with a Node heap limit of about 55% of the memory its
container may use, up to 12 GiB. Native memory, Git, and the sandbox SDKs
use the rest. The AWS CDK worker task (1, 2, or 4 GiB for `small`, `medium`,
`large`) and a Compose host therefore get a limit that fits their memory. No
setting is needed.
To use another limit, set `--max-old-space-size=` in `NODE_OPTIONS` on
the worker. The worker keeps your value. Keep it below the container memory.
The worker also starts with `--heapsnapshot-near-heap-limit=1` and
`--diagnostic-dir=/tmp/heap`. If the heap nears its limit, Node writes one
`.heapsnapshot` file to `/tmp/heap` before the process exits. The file is lost
when the container restarts, unless `/tmp/heap` is on a volume.
Writing the snapshot pauses the worker. It needs extra memory and disk space,
and the file can be larger than the heap. On a small task the write can fail,
and the platform can then stop the process. The result is the same crash as
without the snapshot. To turn the snapshot off, set
`--heapsnapshot-near-heap-limit=0` in `NODE_OPTIONS` on the worker.
The worker logs its heap limit at start (`heapLimitMib` in the
`openworkflow.config-loaded` line).
## Secret handling
Protect and rotate:
* `AUTH_SECRET`
* `CONNECTION_SECRETS_ENCRYPTION_KEY`, if set
* GitHub App private keys and webhook secrets
* Atlassian OAuth client secrets and Forge tokens
* Slack client secret, signing secret, and stored bot-token encryption material
* Linear OAuth client and webhook secrets, or the shared encryption key used to decrypt connection-stored Linear app secrets
* Notion OAuth client and webhook secrets
* Model provider API keys
* SMTP credentials
* OTLP exporter credentials
`AUTH_SECRET` and `CONNECTION_SECRETS_ENCRYPTION_KEY` can affect sessions and
encrypted connector credentials. Plan each rotation carefully. Verify the connectors that
you have after the change.
## Connector readiness
Before inviting users:
* Create or configure at least one GitHub App connection.
* Confirm GitHub webhook delivery reaches the public backend origin.
* If you use Confluence, set up Atlassian OAuth and Forge install or
provisioning.
* If you use Slack, set all three Slack app secrets. Verify the OAuth callback
and the Events Request URL. When you need multi-workspace distribution, test
the installation in a non-development workspace.
* If you use Linear, register a Linear OAuth application. Use the product UI,
or `LINEAR_*` on the backend and the worker.
* Register the exact OAuth callback and webhook URLs.
* Confirm that the backend and the worker can reach `api.linear.app`.
* If you use Notion, set `NOTION_CLIENT_ID` and `NOTION_CLIENT_SECRET`. Register
the exact OAuth redirect URL. Set `NOTION_WEBHOOK_SECRET` after the first
webhook delivery.
* If you use PagerDuty, register the exact OAuth callback.
* Save the client ID and secret in the connector wizard. Or set
`PAGERDUTY_CLIENT_ID` and `PAGERDUTY_CLIENT_SECRET` on the backend and the
worker. ctxpipe creates the webhook subscription after authorization.
* If you use Buildkite, confirm that the backend and worker can reach
`api.buildkite.com`. Confirm that the backend can reach
`graphql.buildkite.com`, which the connect check uses.
* Confirm that the backend and the worker can reach `identity.pagerduty.com` and
`api.pagerduty.com` or `api.eu.pagerduty.com`.
* If you use Plain, confirm that the backend and the worker can reach
`core-api.uk.plain.com`. Plain posts webhooks only to a public HTTPS URL.
Without one, Plain sources use the daily sync and **Sync now**.
* Confirm the worker can run connector jobs.
* Confirm `codesearch` can clone the expected repositories.
* Confirm model provider credentials work for both chat and embedding calls.
## Preflight checklist
* `curl https:///.status` returns JSON with `status: "ok"`.
* Sign-in works using the enabled auth method.
* A user can create or select an organization.
* A repository can be added and reaches indexed state.
* Chat or MCP can answer a question grounded in the repository.
* A Workspace chat answers in a sandbox, and `SANDBOX_PROVIDER` is not
`unsandboxed`. See [Operations](/docs/self-hosting/operations) for the
sandbox checks.
* Backend, worker, and codesearch logs are collected.
* Postgres and graph database backups are configured.
* Upgrade and rollback steps are written down for the current deployment shape.
## Related pages
# Upgrades (https://docs.ctxpipe.ai/docs/self-hosting/upgrades)
An upgrade is an operational change. Back up state. Run migrations once. Deploy
compatible services together. Validate product workflows before you invite users
back in.
## Upgrade checklist
1. Read release notes or the relevant pull request.
2. Back up Postgres.
3. Back up or snapshot the graph database.
4. Keep the current deployment secrets and image tags or commit SHA.
5. Confirm that no connector or ingestion job that runs for a long time must finish first.
6. Run migrations against the target database (AWS CDK: this is part of
`cdk deploy`).
7. Deploy backend, worker, UI, and codesearch from the same commit or release.
8. Validate health, sign-in, repository indexing, Chat/MCP, and connectors.
## Upgrade from main to Workspaces
This release is a minor upgrade. You change the version number and deploy. The SQL only
adds to your database, with one exception. The migration deletes orphan rows
that have no organization (see [What the SQL changes](#what-the-sql-changes)).
Nothing runs on its own after the deploy. Each change to a Workspace
repository and each paid run waits for a click in the UI. Hosted and
self-hosted deployments use the same UI workflow.
### Before you start
1. **Take a database snapshot.** The upgrade deletes orphan rows. A
schema rollback is possible only from a snapshot. Take the snapshot of
Postgres (and of the graph database) before you deploy.
2. **Check the GitHub App permissions.** Workspace writes need **Contents:
Read and write** and **Pull requests: Read and write**. Without them, a
Workspace shows `blocked: write_access`. The upgrade pull request
then cannot open. Update the permissions in the GitHub App settings. Then
accept the change for each installation.
3. **Note your current image tag or release.** You need it for a rollback.
### Deploy
Use the flow for your platform below. Every platform runs the same single
migrate command, `bun run apps/backend/src/db/migrate.ts`.
* **Docker Compose:** pull the new release, then run
`docker compose --profile deploy build` and
`docker compose --profile deploy up -d`. The `migrate` service runs before
the app services. `chat-sandbox-image` builds the new chat image in
`dind`. Set `POSTGRES_PASSWORD` in your `.env` (Compose sends it to migrate
as the `ctxpipe_app` password). Check `docker compose logs migrate`.
Before the Workspaces release, the deploy profile published Postgres on host
port 5433 (`CTXPIPE_POSTGRES_HOST_PORT`). From the Workspaces release, it
publishes no Postgres port. If a backup job or a tool on the host uses port
5433, change it to
`docker compose exec postgres pg_dump -U ctxpipe ctxpipe > backup.sql`.
* **AWS CDK:** run `pnpm update @ctxpipe/aws-cdk`, then `cdk deploy`. See
[the sandbox host](#aws-cdk-sandbox-host) for the cost of the new host.
* **Image-based:** run migrate once with the owner `DATABASE_URL`. Pass the
app password as `DATABASE_APP_PASSWORD`, or make sure the `ctxpipe_app`
role exists. Then start the new images with the `ctxpipe_app` URL.
### What the SQL changes
The upgrade follows the expand-only rule. A minor release adds tables,
columns, policies, and rows. It does not remove anything that the previous
release reads.
* The migration deletes orphan rows (rows with no `org_id`) in
`claim_evidence`, `confluence_spaces`, and `repository_checkouts`. The
delete runs in the same transaction as the migration. If the migration
fails, the rows stay. ctxpipe keeps no copy of these rows.
* `org_id` stays nullable on these tables. Row-level security hides rows with
no `org_id` from the app role.
* A `BEFORE INSERT` trigger on each of the three tables copies `org_id` from
the parent row when it is null. This covers writes from the previous release
after a rollback.
* Every migrate run then calls the backfill, which repairs the other rows
that have no `org_id`. The backfill is safe to run again. You do not run it
by hand.
* The only new column is the nullable `workspaces.format_version`. It caches
the format of the Workspace repository. Git stays the source of truth.
### Removed routes
The old repository list, chat, and knowledge graph pages are gone. A bookmark
to `//repositories`, `//chat`, or `//knowledge-graph` opens the
organization home page. The same data is in the Workspace.
### Connector handover
After the upgrade, each connector binding that a Workspace owns stops writing
to the old repository. The Linear, Notion, Slack, and Confluence connectors
write only to the Workspace. No connector writes the same content to both
places. You do not need to change a connector setting.
### Workspace format file
Each Workspace repository holds `.ctxpipe/workspace.yaml`. It has one key,
`format`, with an integer. ctxpipe reads it at every hydrate and caches it in
`workspaces.format_version`.
| Format | Layout |
| ------ | ------------------------------------------------------------------------------------------- |
| 0 | No file: a repository from before Workspaces, or a foreign repository |
| 1 | Knowledge that ingestion cannot rebuild is exported; sources are in `.ctxpipe/sources.yaml` |
| 2 | One Markdown file for each source under `sources/` |
| 3 | Ingestion v2 ownership front matter (current) |
* Only an upgrade pull request or the bootstrap commit of a new Workspace
writes the file. Bootstrap writes it on an empty repository, or on a
repository with no `.ctxpipe/`, `sources/` or `repositories/` folder below
the Workspace root folder. No agent or chat write can change it.
* A file that does not parse makes the Workspace read-only
(`format_invalid`). Reads still work.
* A format above the one your release knows (for example after a rollback)
makes the Workspace read-only (`format_ahead`). Reads still work.
### Upgrade a Workspace
An upgrade of a Workspace has two parts. These are a git upgrade and a re-ingestion.
Each part needs one click. Nothing starts by itself, unless an owner or admin
turns on [automatic upgrades](/docs/workspaces/settings#automatic-upgrades). A
merge always needs a person. This is true on hosted and on self-hosted
deployments.
#### Where to find the upgrades
* **Workspace settings:** open the Workspace, then **Settings**. The
**Status** group lists each open item. It also shows the format of the
repository, the format that ctxpipe reads, and the automatic-upgrades
setting.
* **Home alert:** the organization home page shows one alert for each
Workspace that has an item to finish. Click **Review upgrade** to open the
Workspace settings. The alert shows nothing when no item is pending.
#### Who can start an item
An organization owner or admin can click **Start**. A member sees the
upgrades but gets **Ask an admin** in place of the button.
#### Item status
Each item shows one status. For the statuses, the next action for each status,
and the reasons for a blocked item, see
[Upgrades](/docs/workspaces/settings#upgrades) in Workspace settings.
If someone closes the pull request without a merge, the item goes back to
**Ready to start** with a note. Click **Start** to open a new pull request.
#### Step 1 to 3: the git upgrade
1. Click **Start** on the git upgrade. ctxpipe opens one pull request for the
Workspace on the branch `ctxpipe/upgrade/format-`. ctxpipe never
pushes to the default branch.
2. The pull request holds one commit for each step that changes a file. The
last commit writes `.ctxpipe/workspace.yaml`, the format file. This is
the format commit.
3. Review the pull request. Then merge it. The merge lands the files and the
new format together.
The steps run in this order:
| Step | Result |
| ---- | -------------------------------------------------------------------------------------------------------------------------------- |
| 1 | Moves the sources into `.ctxpipe/sources.yaml`. Exports the knowledge that fresh ingestion cannot make again. |
| 2 | Writes one Markdown file for each source under `sources/`. |
| 3 | Gives the files that the old pipeline extracted to ingestion v2. Writes the changes that hydrate makes on a format 3 repository. |
**What step 1 keeps.** Step 1 writes this knowledge to the `knowledge/`
folder of the Workspace repository:
* Knowledge from a repository that is not on GitHub.
* Knowledge from a GitHub repository that the sources file does not link.
* Knowledge of the Workspace repository that no Markdown file of the
Workspace holds.
* The teams from CODEOWNERS files.
**What the report lists.** Step 1 does not export other knowledge. The
re-ingestion or hydrate makes it again, or no source reads it. The pull
request description is the report. It lists each item that a step kept, added,
or could not move, for example a connection that maps to no Workspace. Read
the report before you merge.
**A merge is a second approval.** The click on **Start** approves the pull
request. The merge approves the change to your repository. Without a merge,
the format stays the same.
#### Re-ingestion
Fresh ingestion from your source repositories is a standard upgrade step. It
runs ingestion v2 again for each GitHub source repository of the Workspace.
Until you merge the pull request and run it, MCP serves only what the
Workspace already holds.
* **Needs format 3.** The item is blocked until the Workspace is at format 3.
Merge the git upgrade first.
* **Throttle.** One re-ingestion of an organization runs at a time. Its
repositories run one after the other. An instance runs at most four
re-ingestions at once. A run that waits for its turn stays queued and
starts when a slot is free. See
[sandbox limits](/docs/workspaces/chat#sandbox-limits).
* **Spend.** ctxpipe records the model spend of each repository run. It
does not cap the spend. A run does all of its steps. Use the limits of your
model provider key to control spend. When a runtime reports no usage for
a model step, the report shows "Spend unknown". The spend then counts
only the known part.
* **Restart and resume.** A run that a restart of the worker interrupts
continues. If a run stops with no result, a new click on **Start again**
makes a new run. A repository that an earlier run finished at the same
commit is not paid for again.
* **Cancel.** An owner or admin can stop a queued or running re-ingestion
with **Cancel** on its row, or with
`POST /{orgSlug}/api/v1/workspaces/{workspaceSlug}/upgrades/{key}/cancel`.
ctxpipe stops the upgrade run, its repository runs and their ingestion
runs. They start no next step, so they make no new model call and publish
nothing. A step that runs at the cancel can end, but it records nothing.
A publish step pushes only while its run is not canceled, and a cancel
waits until a push that started ends.
The item becomes **Canceled** and gets no automatic retry. **Start again**
makes a new run.
An organization with GitHub data but no Workspace is a new user. It creates
its Workspace in normal onboarding, and its repositories get a fresh
ingestion there.
### Repositories from before Workspaces
Before Workspaces, an organization linked code repositories to the
organization. When the organization moves to Workspaces, ctxpipe links these
repositories to its **first Workspace**. The upgrade pull request of that
Workspace does the work:
* A GitHub repository becomes a GitHub source. After step 2 it is a source file
at `sources/github//.md`.
* A `repositories/*.md` file of a GitHub repository becomes a source file too.
Its `branch` becomes the `ref`. ctxpipe deletes the `repositories/*.md` file in
the same pull request.
* A repository that the Workspace repository already is, or that another
source already links, is not added again.
* A repository that has no GitHub connection becomes a plain git source file at
`sources/git//.md`, when ctxpipe can read its URL as a plain git URL
under the deployment's URL rule. See
[Plain git repositories](/docs/workspaces/linked-repositories#plain-git-repositories)
for the URL rules.
* ctxpipe does not link any other repository. The pull request description names
each one. To add a plain git repository as a source, open a pull request in
Git that adds its source file. Workspace **Settings** lists only GitHub App
repositories. Or leave the repository out.
Below format 3, the Workspace serves the knowledge from before Workspaces as
read-only units from the database (source `legacy`). After the re-ingestion of a
repository, the fresh knowledge replaces the old knowledge of that repository.
#### Index a repository with the REST API
The repository API is from before Workspaces. It indexes a repository for the
organization:
* `POST /{orgSlug}/api/v1/repositories` with `name` and `gitUrl` adds a
repository and queues its indexing. A plain git repository works the same way.
* `POST /{orgSlug}/api/v1/repositories/{id}/reindex` starts its indexing again.
Each index of a repository also updates the knowledge of each Workspace whose
source file names it. A repository is part of a Workspace only when a source
file of that Workspace names it.
### AWS CDK sandbox host
The chat sandbox host is **on by default**. A deploy of this release adds one
EC2 host. The default instance for `size: "small"` is `t4g.medium`
(2 vCPU, 4 GiB, 30 GB Docker volume). It costs about **$28 a month** in
us-east-1. That is the bottom of the $28 to $107 range of the three `size`
profiles. It runs about 6 active sandboxes at once, which is the smallest
instance that meets the sandbox needs. See
[the sandbox host](/docs/self-hosting/deployment/aws#chat-sandbox-host) for
the larger sizes.
To turn the host off, set `sandboxHost: { enabled: false }` on `CtxPipe`. The
deploy then adds no EC2 host and gives the services no sandbox host settings.
Without the host, chat is off. It answers "sandbox unavailable" (503).
## AWS CDK flow
To upgrade a self-hosted AWS stack, update `@ctxpipe/aws-cdk` and deploy again. The published package pins the backend, worker, UI, codesearch, and migrate images to the same git SHA. It also applies the Fargate size profiles of that release.
Before the first deploy of a new version, read the Aurora PostgreSQL minor
version of your cluster. CloudFormation cannot move a cluster to a lower minor
version. A cluster above the pinned version therefore needs that version set in
your CDK app. See
[Aurora PostgreSQL minor version](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#aurora-postgresql-minor-version).
```bash
pnpm update @ctxpipe/aws-cdk
cdk synth
cdk deploy
```
`cdk deploy` runs the migrate task. Then it updates the ECS services. Do not retag services to `:latest` or mix image SHAs from different package versions. See [AWS](/docs/self-hosting/deployment/aws).
The migrate task of this release also creates the `ctxpipe_app` runtime role. The services then read a new runtime secret with the `ctxpipe_app` URL. The earlier `DATABASE_URL` secret keeps the owner URL, so a rollback to the earlier task definitions still works. You do not run `psql` or add `CtxPipe` props. Image-only retags without a package bump skip that role split.
### AWS CDK knowledge store host
The first upgrade to a version with the knowledge store adds one EC2 host for
ClickHouse, an EBS data volume, and an S3 bucket. The host is always on: no
prop turns it off. At `size: "small"` it is a `t4g.large` with a 50 GB volume,
for about **$55 a month** in us-east-1. The deploy waits until ClickHouse
answers. Then the migrate task makes its tables and its two users, before the
services roll. The new knowledge store stays off for each organization. See
[the knowledge store](/docs/self-hosting/deployment/aws#knowledge-store) for
the larger sizes and for recovery.
## Docker Compose flow
For source-based Compose deployments, keep your `docker-compose.override.yml`
(named volumes, extra environment variables, port bindings) in place, then run:
```bash
git fetch origin
git checkout
docker compose --profile deploy build
docker compose --profile deploy up -d
```
`docker compose --profile deploy up -d` also runs `chat-sandbox-image` again.
If the chat image changed, it builds the new image inside `dind`. Then each
conversation gets a new sandbox at its next turn, and continues from its git
branch.
The `deploy` profile runs the `migrate` service before app services. Migrate
connects as the Postgres owner (`ctxpipe` by default) and provisions
`ctxpipe_app`. Backend, worker, and codesearch use the app-role `DATABASE_URL`.
Check the migrator logs before you validate the app.
```bash
docker compose logs migrate
docker compose logs backend worker codesearch
```
## Image-based flow
For platforms that run prebuilt images:
1. Build images for backend, worker, UI, and codesearch from the same commit.
2. Push images with an immutable tag, such as a commit SHA.
3. Run migrations using the backend image for that same commit, with the
**owner** Postgres URL. Then point backend, worker, and codesearch at the
`ctxpipe_app` URL (AWS CDK does this inside `cdk deploy`).
4. Update all app services to the new image tags.
5. Keep the previous image tags available until validation completes.
## Validation
After deployment:
| Area | Validation |
| -------------- | -------------------------------------------------------------------------------------------------------------------- |
| Backend | `GET /.status` returns `status: "ok"` |
| Auth | Sign in with the configured provider |
| UI | Open app through `AUTH_BASE_URL`; do not bypass backend proxy |
| Repositories | Existing repository list loads; new or reindexed repository succeeds |
| Codesearch | Search and file retrieval work for an indexed repository |
| Graph | Knowledge graph page loads for an organization with indexed data |
| MCP | MCP client connects using OAuth |
| Workspace chat | A conversation answers, and its sandbox shows on the Docker daemon (see [Operations](/docs/self-hosting/operations)) |
| Connectors | GitHub webhooks, scoped mirrors, and Slack thread capture still work |
| Observability | Logs and traces continue to reach your configured sinks |
## Rollback
Rollback depends on whether the upgrade ran database migrations.
| Situation | Rollback approach |
| --------------------------------- | ----------------------------------------------------------------------------------------------- |
| No migrations ran | Revert service images or source ref and restart services. |
| Forward-compatible migrations ran | Revert services only if the previous version can tolerate the new schema. |
| Breaking migrations ran | Restore Postgres from backup, restore graph data if needed, then redeploy the previous version. |
### Roll back from Workspaces to main
The SQL of this release is expand-only, so the previous (`main`) image can run
on the upgraded database. Do all of these steps:
1. Put back the previous image tag on every runtime service.
2. **Put back the owner `DATABASE_URL`** on every runtime service. The `main`
image does not set `app.organization_id`. On the `ctxpipe_app` role,
row-level security hides all organization rows, so the app sees no data.
On AWS CDK, deploy the previous `@ctxpipe/aws-cdk` version and restore the
previous runtime secret.
3. Start the services.
After a rollback, the `main` image writes `repository_checkouts`,
`confluence_spaces`, and `claim_evidence` rows with no `org_id`. A trigger
fills `org_id` from the parent row, and the next migrate run repairs the
rest. You do not need to do anything for these rows.
The deleted orphan rows do not come back. Only the snapshot you took before
the upgrade holds them. It is also the only way to roll back the schema.
A Workspace that you upgraded keeps its new format.
A release that knows only older formats shows that Workspace as read-only
(`format_ahead`).
A code rollback without a database restore is not always safe. Take a backup
before you run migrations. Test the restore in an environment like your
production environment.
## Upgrade notes by component
| Component | Watch for |
| -------------- | -------------------------------------------------------------------------------- |
| Backend | Schema migrations, auth behavior, MCP route compatibility, connector API changes |
| Worker | Workflow compatibility, connector job changes, Forge provisioning changes |
| UI | Build-time public URL values, connector wizard behavior |
| Codesearch | Index format or clone/cache behavior |
| Chat sandboxes | Chat image changes, Docker daemon and TLS settings, sandbox host size (AWS) |
| Graph database | Provider configuration and graph projection compatibility |
| Model provider | Default model names, embedding dimensions, provider-specific auth |
## Related pages
# Buildkite (https://docs.ctxpipe.ai/docs/self-hosting/buildkite)
## Overview
A self-hosted ctxpipe uses the same Buildkite connector flow as hosted ctxpipe.
Buildkite has no OAuth application for other products. An organization owner
or admin pastes a Buildkite API access token in the setup. The webhook token is
also connection-specific. ctxpipe encrypts both tokens on the connection with the connection secret key
of the deployment.
There is no `BUILDKITE_*` environment variable. You do not need to change
Compose or CDK settings.
## What you need
* A public HTTPS `AUTH_BASE_URL` for the webhook. Without the webhook, the
connector syncs once a day and on **Sync now**.
* A ctxpipe GitHub App connection with write access to each Workspace repository
that syncs Buildkite builds.
* Network access from the backend and the worker to `api.buildkite.com` over
HTTPS. The backend also reaches `graphql.buildkite.com`, only when someone
connects.
## Webhook URL
Each connection has its own webhook URL, because each webhook has its own
token:
```text
{AUTH_BASE_URL}/api/v1/webhook/buildkite/{connectionId}
```
The webhook step of the setup shows the full URL. ctxpipe verifies the
`X-Buildkite-Signature` HMAC-SHA256 of the timestamp and the raw request body,
and refuses a signature older than five minutes. It also refuses a webhook that
sends the plain `X-Buildkite-Token`. A failed enqueue returns `503`. The next
event in the window or the daily sync then gets the build.
All traffic stays on your deployment. Buildkite sends the webhook to your
`AUTH_BASE_URL`. Your backend and worker call `api.buildkite.com`. When someone
connects Buildkite, your backend also calls `graphql.buildkite.com` once.
## Verify the connector
1. Open **Connectors**, select **Add connection**, then select **Connect** on
the **Buildkite** row.
2. Paste an API access token with only `read_pipelines` and `read_builds`.
Paste the organization slug. ctxpipe refuses a token with any other scope or with
GraphQL API access.
3. Add the webhook in Buildkite with the signature option and the
`build.finished` event, then save its token in the setup.
4. In **Add Buildkite to Workspaces**, select a Workspace and at least one
pipeline. Merge the pull request that adds the source file
under `sources/buildkite/`.
5. Confirm that the source shows **Connected** in the Workspace settings and files appear
under `buildkite/pipelines//`.
6. Finish a build in a selected pipeline. Confirm that
`POST /api/v1/webhook/buildkite/{connectionId}` returns `200`. Confirm that
the build file appears after the 10-minute window.
## Troubleshooting
### Builds do not arrive
* Confirm that Buildkite can reach `AUTH_BASE_URL` over public HTTPS. With a
private URL, no webhook arrives, and the connector syncs once a day and on
**Sync now**.
* A `401` answer has three causes:
* The webhook sends the plain `X-Buildkite-Token`.
* The token in the setup is not the token of this webhook.
* The clock of the backend is more than five minutes off.
Select the signature option in Buildkite. Paste the webhook token again in the
setup. Check the time of the backend host.
* A `503` answer means that ctxpipe could not queue the sync. Read the backend
logs. The next event in the window or the daily sync gets the build.
### The setup cannot check GraphQL access
The setup shows "ctxpipe cannot make sure that the token has no GraphQL API
access. Try again later." when the backend cannot reach
`graphql.buildkite.com`. Allow outbound HTTPS from the backend to that host,
then connect again.
See [Buildkite connector](/docs/connections/source-connectors/buildkite) for
the files, the sync, and product troubleshooting.
# Confluence & Atlassian (https://docs.ctxpipe.ai/docs/self-hosting/confluence-atlassian)
## Overview
Self-hosted Confluence uses the same connector surface as hosted ctxpipe. But
your deployment owns the Atlassian and Forge configuration. There are two
modes:
* **Managed-style install URL**: give a Forge install URL for the connection
or deployment. The wizard then opens Atlassian's install flow.
* **Self-hosted provisioning**: the worker creates, deploys, and installs a
Forge app. It uses a scoped Forge API token and the Confluence site hostname.
Both modes keep connector settings and encrypted credentials on your deployment.
## What you need
* A public `AUTH_BASE_URL` or `CTXPIPE_PUBLIC_APP_URL` that Atlassian can reach.
* A Confluence Cloud site, for example `your-site.atlassian.net`.
* A GitHub connection, and a Workspace that the GitHub App can write to.
* For per-connection OAuth: an Atlassian 3LO app.
* For provisioning: an Atlassian account email and Forge scoped API token.
## 1. Register Atlassian OAuth
If your deployment sets global **`ATLASSIAN_CLIENT_ID`** and
**`ATLASSIAN_CLIENT_SECRET`**, ctxpipe uses that Atlassian app for account linking.
Otherwise, the wizard shows **Register Atlassian OAuth app**. Register a 3LO
app for the Forge connection in the
[Atlassian developer console](https://developer.atlassian.com/cloud/oauth-2-3lo-apps).
Save its client ID and secret in that step. On Docker Compose, the
`ATLASSIAN_*` values need an override file (see
[Docker](/docs/self-hosting/deployment/docker#settings-that-need-an-override-file)).
On AWS CDK, use `connectorSecrets.atlassianClientId` and
`connectorSecrets.atlassianClientSecret`.
Use this callback URL:
`{AUTH_BASE_URL}/api/v1/integrations/atlassian/callback`
ctxpipe encrypts per-connection credentials and does not return the client secret
from connector metadata.
## 2. Install or provision Forge
The wizard resolves the Forge install path in this order:
1. An install URL saved for this Confluence connection.
2. Deployment-wide **`CONFLUENCE_FORGE_INSTALL_URL`**.
3. No install URL, which shows the self-hosted provisioning form.
Provisioning runs `forge register`, `forge deploy`, and `forge install` through
a durable worker job. The generated Forge app uses your public API origin for
its remote callback.
The Forge app requires `read:attachment:confluence` to clone page attachments.
After you upgrade the app manifest that you have, deploy the new Forge version.
Complete Atlassian's permission upgrade. A refresh of the app-system token alone
does not grant the new scope. ctxpipe stores assets in the Workspace repository.
The limits are 25 MiB for each file and 100 MiB for each page. The worker removes
credentials from media redirects. It never saves Atlassian download URLs.
### Allowed Forge apps
ctxpipe accepts a Forge call only from the Forge app of that Confluence
connection. The allowed app is one of these:
* The app ID that the connection saved. Provisioning saves this ID before
`forge install`.
* When the connection has no saved app ID: the app ID in the install URL of the
connection, the app ID in **`CONFLUENCE_FORGE_INSTALL_URL`**, or the hosted
ctxpipe app. The first install event saves the ID of the app.
## 3. Finish the connector
After Atlassian is linked and Forge is installed:
1. Link or confirm the GitHub connector.
2. In **Add to Workspaces**, select the Workspaces that receive Confluence
content.
3. Select the Confluence spaces and pages to ingest.
4. Review and merge the pull request that adds `sources/confluence/.md`
in each Workspace repository.
ctxpipe then syncs Confluence content into each selected Workspace repository.
The content goes through the same indexing and graph ingestion path as other
source material.
## Error codes (provision)
| Code | What to check |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `forge_missing_operator_email` | Give the email of the Atlassian account that owns the Forge token |
| `forge_auth_failed` | Forge scoped API token (App: Forge) at id.atlassian.com |
| `forge_developer_space_noninteractive` | The Atlassian account has no Forge developer space, and the job cannot create one. Create it in the Atlassian developer console, then retry. |
| `forge_developer_space_ensure_failed` | ctxpipe could not find or create the Forge developer space. Retry; contact support with job id |
| `forge_lint_failed` | Bundled app; contact support with job id |
| `forge_deploy_forbidden` | Atlassian user cannot deploy this app |
| `confluence_install_forbidden` | Install requires Confluence admin |
| `confluence_install_site` | Site host / Confluence product |
| `network` | Firewall, DNS, outbound HTTPS |
| `unknown` | Retry; contact support with job id |
## Environment (operators)
* **`AUTH_BASE_URL` / `CTXPIPE_PUBLIC_APP_URL`** - public origin for API and callback URL registration.
* **`ATLASSIAN_CLIENT_ID` / `ATLASSIAN_CLIENT_SECRET`** (optional) - global Atlassian OAuth app for account linking. If unset, use per-connection 3LO in the wizard.
* **`CONFLUENCE_FORGE_INSTALL_URL`** (optional) - deployment-wide fallback for
the **Install** step when a connection does not have its own install URL.
## Related pages
# GitHub (https://docs.ctxpipe.ai/docs/self-hosting/github)
GitHub is the base of a self-hosted ctxpipe deployment. The GitHub App reads the
linked code repositories. It also writes to the Workspace repositories. It
writes the knowledge, the source files, and the content of Linear, Notion,
Confluence, PagerDuty, and Slack.
Your deployment must use a GitHub App that **you create and own**. Self-hosted
repository traffic and credentials do not pass through the hosted ctxpipe GitHub
App.
## Before you start
* Configure a stable public HTTPS `AUTH_BASE_URL`.
* Sign in as an owner or admin of the ctxpipe organization.
* Decide if a GitHub user account or a GitHub organization owns the App.
* Make sure that you can install the App on the GitHub accounts that own the
target repositories.
A GitHub OAuth App used for product sign-in is not the repository GitHub App.
The repository App has its own ID, private key, webhook secret, permissions,
and installation.
## 1. Start the in-product setup
Open **Connectors**, select **Add connection**, then select **Connect** on the
**GitHub** row. The self-hosted wizard reserves a connection and shows the exact
callback, setup, and webhook URLs for that connection.
Keep the wizard open while you register the App. The webhook URL contains a
connection identifier. Copy it from the wizard. Do not guess it.
## 2. Register the GitHub App
Create a [new personal GitHub App](https://github.com/settings/apps/new), or
open a GitHub organization's **Settings → Developer settings → GitHub Apps →
New GitHub App**.
Use the values that the ctxpipe wizard shows:
| GitHub field | Value |
| ---------------------- | --------------------------------------------------- |
| **GitHub App name** | A unique name your operators recognize |
| **Homepage URL** | Your `AUTH_BASE_URL` |
| **Callback URL** | `{AUTH_BASE_URL}/.auth/api/v1/auth/callback/github` |
| **Setup URL** | `{AUTH_BASE_URL}/.github/setup` |
| **Redirect on update** | Enabled |
| **Webhook active** | Enabled |
| **Webhook URL** | Copy the connection-specific URL from the wizard |
| **Webhook secret** | Copy the generated secret from the wizard |
The connection-specific webhook follows this shape:
```text
{AUTH_BASE_URL}/api/v1/webhook/github/
```
## 3. Grant the required permissions
Set repository permissions:
* **Contents:** Read and write
* **Pull requests:** Read and write
* **Issues:** Read-only
Set account permissions:
* **Email addresses:** Read-only
Subscribe to these events:
* **Pull request**
* **Pull request review**
* **Pull request review comment**
* **Issues**
* **Issue comment** (conversation on issues and pull requests)
* **Push**
* **Repository**
The pull request events let ctxpipe mirror the closed pull requests of every
linked repository, merged or not. It writes them into
`github/pulls///` of each Workspace that links the repository. The
**Issues** and **Issue comment** events let ctxpipe mirror the issues of every
linked repository into `github/issues///` in the same way. The App
installation must be able to read those linked repositories. ctxpipe does not
mirror required-check conclusions.
A backfill of the issues of one linked repository reads up to 200 of the most
recently updated issues. It sends one GraphQL request for each 50 issues. Each
request includes the 100 most recent comments of each issue. It writes all the
files in one commit.
### Add the Issues permission to an existing App
An App that you created before the issue mirror does not have **Issues:
Read-only** or the **Issues** event. Do these steps:
1. In the GitHub App settings, set **Issues** to **Read-only** under
repository permissions.
2. Subscribe to the **Issues** event.
3. Save the changes.
4. Tell the owner of each installation of the App to accept the new
permission. GitHub shows the request on the installation settings page of
that account and sends an email to the owner.
Until the owner of an installation accepts the permission, ctxpipe does not
mirror the issues of that installation. The pull request mirror and indexing
continue to work. When the owner accepts, GitHub sends an `installation` event.
ctxpipe then backfills the issues of each linked repository of that account.
After that, ctxpipe mirrors each issue change.
ctxpipe needs write access because it commits to Workspace repositories. It
also opens pull requests for source changes, upgrades, and Workspace chat. Limit
the App installation to the repositories that ctxpipe must read or write.
## 4. Generate and save credentials
After you create the App:
1. Copy its numeric **App ID**.
2. Copy the App slug from its public URL.
3. Generate and download a private key (`.pem`).
4. Return to the ctxpipe wizard.
5. Paste the App ID and slug, then upload or paste the complete PEM.
6. Save the connection.
ctxpipe encrypts the private key and webhook secret for this GitHub connection.
Keep `AUTH_SECRET` stable. Use
`CONNECTION_SECRETS_ENCRYPTION_KEY` when your secret-management policy needs an
explicit connector encryption key.
## 5. Install and verify the App
Install the App on the GitHub user or organization that owns the source
repositories. Choose **Only select repositories** unless the deployment
needs broad access.
ctxpipe attaches an installation to an organization only with proof that it
belongs there. A connection that you create in the wizard gives this proof with
its own App credentials. An installation of an App that you configure with
`GITHUB_APP_ID` is different. The person who finishes setup must show that
their GitHub account can see the installation. If their linked GitHub account
cannot show this, ctxpipe asks them to authorize the App once. ctxpipe uses
the short-lived token of that authorization for this check only. It does not
keep the token and does not link the GitHub account.
The authorization needs the client ID and the client secret of that same App
in `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET`. These variables also turn on
GitHub sign-in. The App must list the callback URL
`{AUTH_BASE_URL}/.auth/api/v1/auth/callback/github`. Without these values, use
the wizard.
Then verify the full path:
1. The GitHub connector reaches **Connected**.
2. **Create workspace** lists repositories from the expected installation.
3. A linked repository reaches **Indexed** in the Workspace's **Settings**.
4. Changing repository selection returns through the ctxpipe setup URL.
5. A test push reaches the connection-specific webhook.
6. A source change can open a pull request on the Workspace repository.
## Troubleshooting
### Installation returns to the wrong host
Check the App's Callback URL and Setup URL against the current
`AUTH_BASE_URL`. Keep browser, auth, callbacks, API, and MCP on the same public
backend origin.
### Setup asks to connect a GitHub account
ctxpipe attaches the installation through the App that `GITHUB_APP_ID`
configures. Link a GitHub account that can see the installation, or create
the connection with the self-hosted wizard so its own App credentials prove the
installation.
### Repositories do not appear
Confirm that the App is installed on the correct GitHub account and has access
to the repository. A change to the installation in GitHub does not tell
ctxpipe to index a repository. After that, link the repository from the
Workspace's **Settings**.
### Webhooks fail signature verification
The secret in the GitHub App settings must match the secret saved for this ctxpipe
connection. Do not reuse a connection-specific URL with credentials from
another connection.
### Issues do not appear in the Workspace
Confirm that the App has **Issues: Read-only** and that it subscribes to the
**Issues** event. Then find the cause in the backend logs:
* `github_issue_mirror_no_permission`: the owner of the installation did not
accept the Issues permission. Tell the owner to accept it on the installation
settings page in GitHub. ctxpipe then backfills the issues.
* `github_issue_mirror_repository_not_covered`: the installation does not have
access to the repository. Add the repository to the repository access of the
installation in GitHub. After that, ctxpipe mirrors each issue change of
that repository.
### Connector writes fail
Confirm that **Contents** and **Pull requests** are both read and write.
Confirm that the installation includes the Workspace repository.
# Linear (https://docs.ctxpipe.ai/docs/self-hosting/linear)
Self-hosted Linear uses an OAuth application that you create in Linear. The
usual path is to paste the client ID, client secret, and webhook signing secret
into the connector wizard. That path needs no `LINEAR_*` environment variables
and no service restart.
The same routes run on hosted ctxpipe and on self-host. Hosted ctxpipe keeps a
deployment-wide Linear app in environment variables as a fallback. Self-host
operators can use that fallback too. Use it when you want one shared Linear
application for the whole deployment.
The worker decrypts the saved client secret with the same key the backend uses
(`AUTH_SECRET` or `CONNECTION_SECRETS_ENCRYPTION_KEY`). That is not a new Linear
environment variable.
## What you need
* A public HTTPS `AUTH_BASE_URL`. In production, keep
`CTXPIPE_PUBLIC_APP_URL` on the same origin.
* Permission to create a Linear OAuth application.
* A ctxpipe GitHub App connection with write access to the repository of each
Workspace that uses Linear.
* Network access from the backend and the worker to `api.linear.app` over
HTTPS.
## Connector state and backups
You can review the Linear scope in Git:
* Each Workspace has a source file at `sources/linear/.md`. Draft scope is
the file on the branch of the pull request that adds it.
* Live scope is the same file on the default branch after the merge.
* The deployment stores encrypted credentials and setup status on the
`connections` row. The connection holds authorization only.
* Signed Linear events update in-scope generated files through durable worker
jobs.
Back up PostgreSQL for connection credentials. Back up the Workspace
repositories for both scope and mirrored content. PostgreSQL contains only
credentials and setup state.
## 1. Create a Linear application
Open [Create new application](https://linear.app/settings/api/applications/new)
in Linear Settings → API.
Fill in **Application name** and **Developer name**. In **Redirect URIs**, paste:
```text
https:///api/v1/integrations/linear/callback
```
Linear supports several redirect URIs, one per line. Add local, staging, or
preview callbacks individually when those environments must run OAuth. The
scheme, host, path, and trailing slash must match the URI that ctxpipe sends.
ctxpipe requests the **`read`** scope with a user actor on the authorize URL. You
do not set those on this Linear form.
Linear cannot deliver webhooks to a private localhost address. To test a
connector end to end, use a public HTTPS deployment or tunnel.
## 2. Configure webhooks on the same form
Turn **Webhooks** on and set **Webhook URL** to:
```text
https:///api/v1/webhook/linear
```
Copy **Webhook signing secret** from that form. Linear fills it in before you
select Create. ctxpipe verifies the `Linear-Signature` HMAC against the raw
request body. It rejects stale timestamps. For an accepted in-scope event,
ctxpipe starts a durable update before the endpoint acknowledges delivery.
Under **Data change events**, select Issues, Comments, Documents, Projects,
Initiatives, Teams, and the related update and attachment boxes (including
cycles, issue labels, users, and customer requests). Under **Authorization events**,
select OAuth authorization events.
Then select **Create**. Only after that, Linear shows **Client ID** and
**Client secret**. Copy them for the next step.
A Linear application has one webhook destination. You can register several
OAuth redirect URIs, but events from every workspace installation go to that
one webhook URL. Use a separate Linear application when a staging or PR
environment must receive its own incremental events. Per-organization apps
each register the same Webhook URL and store their own signing secret on the
connection.
## 3. Register the app in the product UI
1. Open **Connectors**, select **Add connection**, then select **Connect** on
the **Linear** row.
2. On **Register Linear OAuth app**, paste the client ID and the client secret
that Linear shows after Create. Also paste the webhook signing secret from the
create form. The wizard shows the exact **Redirect URIs** and **Webhook URL** values
for this deployment.
3. Save the app, then continue to **Connect Linear**.
4. Complete OAuth. In **Add to Workspaces**, choose the Workspaces and the
scope, then merge the pull request that adds `sources/linear/.md`.
ctxpipe stores secrets encrypted on the Linear `connections` row. The list and
status APIs never return them.
The backend and worker must share the encryption key
(`AUTH_SECRET` / `CONNECTION_SECRETS_ENCRYPTION_KEY`) so token refresh can use
the saved client secret.
## 4. Optional deployment-wide environment variables
`LINEAR_*` is an optional operator shortcut for a deployment-wide Linear
application (the hosted fallback). When you set both `LINEAR_CLIENT_ID` and
`LINEAR_CLIENT_SECRET`, the wizard skips Register and starts at Connect.
| Variable | Required | Purpose |
| ----------------------- | ---------------------------------: | ------------------------------------------------------------------------------------------ |
| `LINEAR_CLIENT_ID` | Optional; backend + worker if used | Shared OAuth application client ID |
| `LINEAR_CLIENT_SECRET` | Optional; backend + worker if used | Shared OAuth code exchange and token refresh |
| `LINEAR_REDIRECT_URI` | Backend; optional | Exact callback override; defaults to `{AUTH_BASE_URL}/api/v1/integrations/linear/callback` |
| `LINEAR_WEBHOOK_SECRET` | Optional; backend if used | Shared signing secret for `/api/v1/webhook/linear` |
Empty values still parse as unset. Restart or redeploy both services after
changing these variables.
### Docker Compose
Add the variables to the root `.env` used by the `deploy` profile only if you
want a deployment-wide app. Then run:
```bash
docker compose --profile deploy up -d --build backend worker
```
### AWS CDK
Pass the four values through `connectorSecrets` when you want the
deployment-wide fallback:
```ts
new CtxPipe(stack, "CtxPipe", {
// other required settings
connectorSecrets: {
linearClientId: cdk.SecretValue.secretsManager("linear/client-id"),
linearClientSecret: cdk.SecretValue.secretsManager("linear/client-secret"),
linearWebhookSecret: cdk.SecretValue.secretsManager("linear/webhook-secret"),
// Optional when AUTH_BASE_URL already produces the registered callback:
linearRedirectUri: cdk.SecretValue.unsafePlainText(
"https://app.example.com/api/v1/integrations/linear/callback",
),
},
});
```
The construct stores these values in the connector secret and injects them into
backend and worker task definitions.
## 5. Verify the connector
1. Confirm `GET /.status` succeeds on the public backend origin.
2. Confirm the worker is running and shares the backend encryption key.
3. Open **Connectors**, select **Add connection**, then select **Connect** on
the **Linear** row.
4. If `LINEAR_*` is unset, complete **Register Linear OAuth app**, then Connect.
5. Complete OAuth and confirm the expected Linear workspace appears.
6. In **Add to Workspaces**, choose a Workspace and a scope.
7. Merge the pull request that adds `sources/linear/.md`.
8. Confirm the GitHub App delivers the merge push to
`{AUTH_BASE_URL}/api/v1/webhook/github/`.
9. Confirm the connector reaches **Connected** and generated files appear under
`linear/` in the Workspace repository.
10. Edit a mirrored Linear issue and confirm
`POST /api/v1/webhook/linear` succeeds and the corresponding Git file
changes.
11. Confirm the Workspace re-indexes.
ctxpipe copies Linear file uploads and explicit embedded external media into
sibling `assets/` directories in the Workspace repository. Ordinary link-only
attachments stay links. ctxpipe never crawls them. GitHub pull requests and
commits stay references. The limits are 25 MiB for each asset and 100 MiB for
each Linear entity. An unsafe, failed, or oversized asset leaves a Linear link
or a textual stub. Give the destination repository the same sensitivity as the
selected Linear scope.
Initial sync proves OAuth, worker, and GitHub access. The webhook step
separately proves Linear webhook delivery and signature configuration.
## Troubleshooting
### “Linear OAuth is not configured”
Neither the connection nor the deployment has a Linear client ID and secret.
Register the app in the wizard, or set `LINEAR_CLIENT_ID` and
`LINEAR_CLIENT_SECRET` on the backend and the worker if you want a deployment-wide app.
### Linear reports a redirect URI mismatch
Compare the authorize request's `redirect_uri` with the application setting
character for character. If you can, leave `LINEAR_REDIRECT_URI` unset, so that ctxpipe
derives it from the public `AUTH_BASE_URL`. Use the override only behind a proxy
with a different external callback.
### OAuth succeeds but sync later fails
The worker must decrypt the saved client secret. It needs the same
`AUTH_SECRET` / `CONNECTION_SECRETS_ENCRYPTION_KEY` as the backend. When you
use the environment variable fallback, the worker needs `LINEAR_CLIENT_ID` and
`LINEAR_CLIENT_SECRET`. Check outbound HTTPS and the worker logs. Check if
someone revoked the application authorization in Linear.
### The source pull request cannot open
If **Add to Workspaces** reports an error, confirm that the selected GitHub App
installation still has write access to the repository of the Workspace. Also
confirm that the Workspace repository uses format 2 or higher. Below that, the
dialog shows **Needs the format upgrade first.** next to the Workspace.
### Webhook endpoint returns 503
No webhook signing secret is present on the matching Linear connection or in
`LINEAR_WEBHOOK_SECRET`.
### Webhook endpoint returns 401
The signing secret does not match the Linear application that sent the event, a
proxy changed the raw body, or the event timestamp is stale. Do not parse and
re-serialize the body in a gateway before forwarding it. A secret saved on one
connection does not authorize events for another.
### Initial sync works but later edits do not appear
This is a webhook problem, not an OAuth problem. Confirm that webhooks are
enabled on the application, the public URL is reachable, the relevant resource
categories are selected, and requests reach `/api/v1/webhook/linear`.
## User guide
See [Linear connector](/docs/connections/source-connectors/linear) for scope,
repository layout, review workflow, and user-facing recovery steps.
# Notion (https://docs.ctxpipe.ai/docs/self-hosting/notion)
## Overview
Self-hosted Notion uses the same connector routes as hosted ctxpipe. The
main path is to register a **private Notion OAuth app** in the product UI. Then
you can connect a workspace without `NOTION_*` environment variables and
without a restart. Environment variables stay as an optional shortcut for hosted
and operator setups.
Traffic stays on **this** deployment's `AUTH_BASE_URL`. Do not point OAuth
callbacks or webhook URLs at hosted ctxpipe.
## 1. Register the OAuth app in the UI
1. Open **Connectors**, select **Add connection**, then select **Connect** on
the **Notion** row. ctxpipe creates a draft Notion connection.
2. On **Register Notion OAuth app**, create an OAuth app in the
[Notion integrations portal](https://www.notion.so/profile/integrations).
For a private install, choose **Selected workspaces only**. Do not list the app
on the Marketplace. An internal (token-only) integration does not work.
3. Copy the callback URL from the wizard into the app's OAuth / redirect URI
field. It is:
```text
{AUTH_BASE_URL}/api/v1/connectors/notion/oauth/callback
```
Set **`NOTION_REDIRECT_URI`** only when the public callback differs from the
URL that ctxpipe derives from `AUTH_BASE_URL`.
4. Paste the **Client ID** and **Client secret**, then save. ctxpipe stores the
secret encrypted on that connection. It never shows the secret again.
5. On **Add the webhook**, open the **Webhooks** tab of the same app. Add a webhook.
Paste the wizard event URL into **Webhook URL**. Select the page, data
source, and database event types (`page.*`, `data_source.*`, and
`database.*`). ctxpipe ignores other events. If you do not select a type, ctxpipe
does not get its updates. Query parameters scope the URL to this connection:
```text
{AUTH_BASE_URL}/api/v1/webhook/notion?connectionId={id}&provisioningToken={hmac}
```
The provisioning token is an HMAC of the saved client secret. Notion's first
verification request is unsigned. ctxpipe checks that HMAC. It stores the
verification token on the same row and shows it in the wizard. Paste that
**Verification token** into Notion's **Verify subscription** dialog. Then select
**Verify subscription**.
6. Choose **Continue**, then **Connect Notion**. Then authorize the workspace.
The worker reads the OAuth client credentials from the connection row. Token
refresh therefore works when the `NOTION_*` environment variables are empty.
## 2. Optional env shortcut
Operators who have a shared OAuth app can set
**`NOTION_CLIENT_ID`**, **`NOTION_CLIENT_SECRET`**, and
**`NOTION_WEBHOOK_SECRET`** on the backend and worker instead of saving them
in the wizard. When you set those environment variables, **Add Notion** skips the
register step and opens the OAuth popup.
For webhook provisioning with environment variables only (no `connectionId` on
the webhook URL), generate the provisioning token from the repository root:
```bash
NOTION_CLIENT_SECRET='your-client-secret' bun -e 'import { createHmac } from "node:crypto"; process.stdout.write(createHmac("sha256", process.env.NOTION_CLIENT_SECRET ?? "").update("ctxpipe:notion-webhook-provisioning:v1").digest("base64url"))'
```
Use the result in:
```text
{AUTH_BASE_URL}/api/v1/webhook/notion?provisioningToken={generated-token}
```
Copy the **verification token** from Notion after the first delivery. Set
**`NOTION_WEBHOOK_SECRET`** to it. Provisioning with environment variables only
does not save that token on a connection row.
Do not ask each ctxpipe organization to create a second webhook path. One
webhook URL for each integration is enough. A self-hosted deployment scopes it
with query parameters.
## 3. Supply env secrets when you use the shortcut
### Docker Compose
Add the values to the root `.env`, then restart the backend and the worker:
```bash
NOTION_CLIENT_ID=...
NOTION_CLIENT_SECRET=...
NOTION_WEBHOOK_SECRET=...
pnpm start
```
### AWS CDK
Pass `notionClientId`, `notionClientSecret`, and `notionWebhookSecret` through
the construct's `connectorSecrets`. Keep the values in your CDK secret source.
Do not commit them in stack code.
## 4. Complete connector setup
After you register the integration in the UI or with environment variables:
1. Authorize the Notion workspace.
2. In **Add to Workspaces**, select the Workspaces that receive Notion
content.
3. Select the Notion pages and databases to mirror.
4. Review and merge the pull request that adds `sources/notion/.md`.
ctxpipe writes pages as Markdown under `notion/pages/`. It writes databases
under `notion/databases/`. Each database has one `table.csv` snapshot, and each
row has one Markdown file, for task and knowledge retrieval. ctxpipe copies
provider-declared files and explicit embedded external media into an `assets/`
directory beside the page or row that uses them. The limits are 25 MiB for each file and
100 MiB for each page or row. ctxpipe never saves Notion URLs that expire. A
failed, unsafe, or oversized download leaves a Notion page link or a caption
stub.
## Verify the complete flow
1. Add Notion and save a private OAuth app (or confirm that the environment variables are set).
2. Authorize a clean Notion workspace through the product flow.
3. Share a page or database with the integration and confirm it appears.
4. Select a Workspace in **Add to Workspaces** and merge the pull request that
adds `sources/notion/.md`.
5. Confirm the initial mirror writes readable files under `notion/`.
6. Edit an in-scope source and confirm the deployment webhook receives the
event.
7. Confirm the generated git file changes and the Workspace re-indexes.
## Troubleshooting
### OAuth returns to the wrong deployment
The integration callback must match the public `AUTH_BASE_URL` exactly, including
scheme, host, and path. Set `NOTION_REDIRECT_URI` only when the external route
differs.
### Register step does not appear
The wizard skips registration when both `NOTION_CLIENT_ID` and
`NOTION_CLIENT_SECRET` are set. That is the hosted/operator shortcut.
### Signed events return `401` or `503`
If you used the UI webhook URL, confirm Notion's verification completed on that
connection (the webhook URL includes `connectionId`). If you used the
environment variable shortcut, set `NOTION_WEBHOOK_SECRET` to the verification token shown after
Notion's first delivery, then restart the backend. The provisioning token in
the webhook URL does not replace the signing secret.
### Content sync cannot refresh authorization
Confirm the connection still has an OAuth app (or that both backend and worker
receive the same Notion client ID and secret) and can reach `api.notion.com`.
### No pages appear
Notion returns only the pages and databases that you share with the
integration. Share the resource in Notion. Then refresh resources in the ctxpipe
setup flow.
## User guide
See [Notion connector](/docs/connections/source-connectors/notion) for scope,
repository layout, review workflow, and user-facing recovery steps.
## Related pages
* [Workspaces](/docs/workspaces)
* [GitHub connector](/docs/connections/source-connectors/github)
* [Indexing](/docs/workspaces/indexing)
# PagerDuty (https://docs.ctxpipe.ai/docs/self-hosting/pagerduty)
## Overview
Self-hosted PagerDuty uses the same connector flow as hosted ctxpipe. Someone
must still create a Scoped OAuth application in the
[PagerDuty developer portal](https://developer.pagerduty.com). The usual path
is to paste the client ID and secret of that app in the connector wizard.
ctxpipe then creates one v3 webhook subscription for each connection. Operators
do not paste a shared webhook signing secret.
Deployment environment variables (`PAGERDUTY_CLIENT_ID` /
`PAGERDUTY_CLIENT_SECRET`) are an optional shortcut, as on hosted ctxpipe. You do not need to restart Compose
or CDK to finish first-time setup.
## What you need
* A public HTTPS `AUTH_BASE_URL`. In production, keep
`CTXPIPE_PUBLIC_APP_URL` on the same origin.
* Permission to create a PagerDuty OAuth application.
* A ctxpipe GitHub App connection with write access to each Workspace repository
that syncs PagerDuty incidents.
* Network access from the backend and the worker to `identity.pagerduty.com`
and `api.pagerduty.com` or `api.eu.pagerduty.com` over HTTPS.
PagerDuty cannot deliver webhooks to a private localhost address. To test a
connector end to end, use a public HTTPS deployment or tunnel.
## 1. Register the OAuth app in the wizard
1. Open **Connectors**, select **Add connection**, then select **Connect** on
the **PagerDuty** row. ctxpipe creates a draft connection so that you can
save the app before account authorization.
2. Open [PagerDuty App Registration](https://developer.pagerduty.com/my-apps),
select **New App**. Name the app `ctxpipe` or use the name of your
deployment.
3. Under **Functionality**, select **OAuth 2.0**. Leave **Events Integration**
unselected. ctxpipe creates a v3 webhook subscription through PagerDuty's
API after account authorization. This is not an Events API integration.
4. Choose **Scoped OAuth**. Register this redirect URL exactly. The wizard
copies it for you:
```text
{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback
```
5. Grant only these permission scopes:
* `incidents.read`
* `services.read`
* `users.read`
* `webhook_subscriptions.read`
* `webhook_subscriptions.write`
6. Select **Register App**. Copy the generated client id and client secret
before you close the confirmation. PagerDuty shows the secret only once.
7. Save both values in **Register PagerDuty OAuth app**. ctxpipe
encrypts the secret on the connection. It never returns the secret from
connector metadata. Continue to **Connect PagerDuty**.
Do not add write-back scopes for incidents. ctxpipe does not acknowledge or
resolve PagerDuty incidents.
ctxpipe creates webhook signing secrets for each subscription and stores them
encrypted on the connection. Do not create a deployment-wide PagerDuty webhook
secret.
## 2. Optional: deployment env shortcut
If you want one shared app for the whole deployment, set
**`PAGERDUTY_CLIENT_ID`** and **`PAGERDUTY_CLIENT_SECRET`** on the backend and
the worker. Set **`PAGERDUTY_REDIRECT_URI`** only when the public callback
differs from the URL that ctxpipe derives from `AUTH_BASE_URL`. The wizard then starts at
**Connect PagerDuty**.
ctxpipe looks for credentials in the saved connection app first. Then it looks
in the environment variables. The worker needs the client secret that the
refresh path reads.
### Docker Compose
Add the values to the root `.env`, then restart the backend and the worker:
```bash
PAGERDUTY_CLIENT_ID=...
PAGERDUTY_CLIENT_SECRET=...
pnpm start
```
### AWS CDK
Pass the values through `connectorSecrets`:
```ts
new CtxPipe(stack, "CtxPipe", {
// other required settings
connectorSecrets: {
pagerdutyClientId: cdk.SecretValue.secretsManager("pagerduty/client-id"),
pagerdutyClientSecret: cdk.SecretValue.secretsManager(
"pagerduty/client-secret",
),
// Optional when AUTH_BASE_URL already produces the registered callback:
pagerdutyRedirectUri: cdk.SecretValue.unsafePlainText(
"https://app.example.com/api/v1/integrations/pagerduty/callback",
),
},
});
```
The construct stores these values in the connector secret and injects them into
backend and worker task definitions.
## 3. Public webhook URL
After a user completes OAuth, ctxpipe creates one account-level v3 webhook
subscription for incident events and points it at:
```text
{AUTH_BASE_URL}/api/v1/webhook/pagerduty
```
Do not paste this URL into **Events Integration** during App Registration.
PagerDuty receives it when ctxpipe creates the webhook subscription through
the Webhook Subscriptions API.
ctxpipe verifies `X-PagerDuty-Signature` HMAC-SHA256 against the raw request
body. It binds the event to the one connection whose stored secret verifies. It
routes with `X-PagerDuty-Subscription`. Re-authorization replaces a lost
secret instead of stacking another subscription on the same event URL.
Account-level delivery can include every service. ctxpipe syncs an event only
for the Workspace sources whose services include the incident's service.
ctxpipe acknowledges events with no service, or with a service that no source
selects, with `200`. It never fetches them.
## 4. Verify the connector
1. Confirm `GET /.status` succeeds on the public backend origin.
2. Open **Connectors**, select **Add connection**, then select **Connect** on
the **PagerDuty** row.
3. If the wizard shows **Register PagerDuty OAuth app**, save the client id
and secret. Skip this when the deployment already has a shared app that environment
variables set.
4. Complete OAuth and confirm the expected account appears.
5. In **Add PagerDuty to Workspaces**, select one or more Workspaces and at
least one service for each. Use search. Show more services if the account
has many.
6. Merge the pull request that adds `sources/pagerduty/.md` in each
Workspace repository.
7. Confirm the source shows **Connected** in the Workspace settings and generated
files appear under `pagerduty/services//incidents/`.
8. Trigger or update an in-scope incident and confirm
`POST /api/v1/webhook/pagerduty` succeeds and the corresponding Git file
changes.
9. Confirm the Workspace re-indexes.
## Upgrade from `pagerduty/config.yaml`
Earlier versions kept the service scope in `pagerduty/config.yaml` in one sync
repository. Upgrade step 1 of each Workspace moves the connection to Workspace
sources. An org owner or admin starts it with **Start** in the
Upgrades section of the Workspace settings. You can also start it from the
command line:
```bash
pnpm --filter @ctxpipe/backend migrate-workspace-sources --org
pnpm --filter @ctxpipe/backend migrate-workspace-sources --org --apply
```
The first command only prints the plan. With `--apply`, the command starts
the Workspace upgrade, as the button does. The upgrade opens one pull request
for each Workspace and never pushes to the default branch. When the Workspace
links the old sync repository, the pull request adds the merged services as a
source. The old per-connection sync stops after the merge, when the new source
syncs. When the Workspace repository is the old sync repository, the same pull
request moves each file in `pagerduty/incidents/` to
`pagerduty/services//incidents/`, with its assets. The service id
comes from the `service_id` front matter of the file. Git keeps the history of
each moved file. A file without a readable `service_id` stays in
`pagerduty/incidents/`.
## US and EU accounts
ctxpipe finds the account region from the authorized user's `html_url`
after OAuth. Do not add a region field to the wizard. The API host is
`api.pagerduty.com` or `api.eu.pagerduty.com`.
## Troubleshooting
### “PagerDuty OAuth is not configured”
Neither the connection nor the deployment has OAuth client credentials. Finish
**Register PagerDuty OAuth app** in the wizard, or set both
`PAGERDUTY_CLIENT_ID` and `PAGERDUTY_CLIENT_SECRET` on the backend and the
worker and restart. The catalog still opens the setup wizard. Connect fails when
ctxpipe cannot create the authorization URL.
### Callback is rejected
The registered redirect URI must match
`{AUTH_BASE_URL}/api/v1/integrations/pagerduty/callback` or
`PAGERDUTY_REDIRECT_URI` exactly, including scheme, host, path, and whether a
trailing slash is present.
### Incremental events never arrive
Confirm these three points: the public webhook URL is reachable, the connection
is connected, and the service of the incident is in a **Connected** PagerDuty source
of a Workspace. ctxpipe rejects tampered or stale signatures. A failed enqueue
returns `5xx`, so PagerDuty can retry.
### A source shows “Out of access”
Someone deleted a selected service in PagerDuty, or the OAuth grant cannot read
it. ctxpipe tries the service in three sync cycles and keeps its files. Then it
marks only that source. Remove the service from the source, or give the
connection access to it.
See [PagerDuty connector](/docs/connections/source-connectors/pagerduty) for
scope, files, and product troubleshooting.
# Plain (https://docs.ctxpipe.ai/docs/self-hosting/plain)
## Overview
Self-hosted Plain uses the same connector flow as hosted ctxpipe. Plain has no
OAuth application for other products. An organization admin pastes the API key of a
Plain machine user in the setup. ctxpipe encrypts the key and the webhook
token on the connection with the connection secret key of the deployment.
There is no `PLAIN_*` environment variable. You do not need to change Compose
or CDK settings.
## What you need
* A public HTTPS `AUTH_BASE_URL` for the webhook target. Plain sends webhooks
only to a public HTTPS address. Without it, the connector syncs once a day
and on **Sync now**. ctxpipe cannot check that Plain reaches the URL. When
Plain accepts the target but cannot reach it, the card still shows live
updates as on.
* A ctxpipe GitHub App connection with write access to each Workspace repository
that syncs Plain content.
* Network access from the backend and the worker to `core-api.uk.plain.com` over HTTPS.
## Webhook target
At connect, ctxpipe creates one Plain webhook target for each connection:
```text
{AUTH_BASE_URL}/api/v1/webhook/plain/{connectionId}
```
The target sends `timeline.timeline_entry_changed` events with a random token
in the `X-Ctxpipe-Webhook-Token` header. ctxpipe compares the token and
refuses a request without it. A failed enqueue returns `503`. Plain then sends
the event again.
## Verify the connector
1. Open **Connectors**, select **Add connection**, then select **Connect** on
the **Plain** row.
2. Paste the API key of a machine user with the permissions that the setup
lists.
3. In the **Add to Workspaces** step, select a Workspace and at least one
Help Center or label. Merge the pull request that adds the source file under
`sources/plain/`.
4. Confirm the source shows **Connected** in the Workspace settings and files appear
under `plain/`.
5. In Plain, open **Settings → Webhooks** and confirm the ctxpipe target. Add a
note to a real thread with a selected label. Do not use a Plain test thread.
Confirm that `POST /api/v1/webhook/plain/{connectionId}` returns `200`.
Confirm that the thread file changes after the 10-minute window.
See [Plain connector](/docs/connections/source-connectors/plain) for the
files, the sync, and product troubleshooting.
# Slack (https://docs.ctxpipe.ai/docs/self-hosting/slack)
## Overview
Self-hosted Slack uses the same connector flow as hosted ctxpipe. But your
deployment owns **one** Slack app. Organizations authorize that app into their
workspace. They do not create their own Slack apps or webhook URLs.
This page covers only the operator setup. It has the Slack app, its scopes and
events, and the secrets. For how people capture threads, see the
[Slack connector guide](/docs/connections/source-connectors/slack).
## What you need
* A public HTTPS `AUTH_BASE_URL` that Slack can reach.
* A Slack app with OAuth and Event Subscriptions enabled.
* A GitHub connection, and a Workspace that the GitHub App can write to.
* Backend environment variables: `SLACK_CLIENT_ID`, `SLACK_CLIENT_SECRET`, `SLACK_SIGNING_SECRET`.
* Model environment variables (`MODEL_PROVIDER*` / Bedrock), so the mention
agent can read extra text after `@mention`. A bare mention captures without
the model.
* The same stable `AUTH_SECRET` and, when set,
`CONNECTION_SECRETS_ENCRYPTION_KEY` on backend and worker so the worker can
decrypt stored bot tokens.
## 1. Create the Slack app
In the [Slack API dashboard](https://api.slack.com/apps), create an app. You can
start from scratch. For higher rate limits in production, distribute it as a
Marketplace app or an internal organization app. Do not use a throwaway development app.
If organizations will install the app into workspaces other than the app
owner's development workspace, activate **Manage Distribution** in Slack.
Marketplace listing is a separate step. Private distribution to many
workspaces does not need it.
### OAuth & Permissions
Add at least these bot token scopes:
* `app_mentions:read` (receive capture requests from bot mentions)
* `channels:history`, `channels:read`
* `groups:history`, `groups:read` (private channels; invite the bot)
* `chat:write` (in-thread capturing → captured status replies)
* `files:read`
* `users:read`
If you add a scope after a workspace installs the app, that workspace must
complete OAuth again. A change to the Slack app settings does not upgrade the
bot token that you have.
Register this OAuth redirect URL:
```text
{AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback
```
Set **`SLACK_CLIENT_ID`** and **`SLACK_CLIENT_SECRET`** on the backend from the
app credentials. Set **`SLACK_REDIRECT_URI`** only when the public callback
differs from the URL derived from `AUTH_BASE_URL`.
### Event Subscriptions
Enable Event Subscriptions and set the Request URL to:
```text
{AUTH_BASE_URL}/api/v1/webhook/slack
```
Slack verifies the URL with a signed `url_verification` challenge. ctxpipe checks
`X-Slack-Signature` using **`SLACK_SIGNING_SECRET`**.
Subscribe the bot to:
* **`app_mention`**: the capture requests;
* **`app_uninstalled`** and **`tokens_revoked`**: ctxpipe marks the connection
revoked when a workspace removes the app.
Do **not** subscribe to `message.channels` / `message.groups` for the connector
ingest path. Continuous message fan-out is not used.
**Turn Socket Mode off.** If Socket Mode is enabled, Slack does not POST to the
Request URL (“You won’t need a Request URL”) and mentions never reach ctxpipe.
The Events URL is deployment-wide. Do not ask each organization to register its
own webhook.
## 2. Wire deployment secrets
Set all three Slack values together. A partial configuration leaves OAuth or
event verification unusable.
### Docker Compose
Add the values to the root `.env` used by the `deploy` profile, then recreate
the backend and the worker:
```dotenv
SLACK_CLIENT_ID=...
SLACK_CLIENT_SECRET=...
SLACK_SIGNING_SECRET=...
```
```bash
docker compose --profile deploy up -d --force-recreate backend worker
```
### AWS CDK
Pass Slack credentials through `connectorSecrets`:
```ts
new CtxPipe(stack, "CtxPipe", {
// ...required props
connectorSecrets: {
slackClientId: cdk.SecretValue.secretsManager("slack-client-id"),
slackClientSecret: cdk.SecretValue.secretsManager("slack-client-secret"),
slackSigningSecret: cdk.SecretValue.secretsManager("slack-signing-secret"),
},
})
```
Use your normal secret-management convention. Do not put plaintext credentials
in source control.
## 3. Check the connector
After the Slack app is configured, an organization admin connects Slack on
**Connectors**, adds the channels to a Workspace, and merges the source pull
request. See the [Slack connector guide](/docs/connections/source-connectors/slack).
* One Slack workspace connects to one ctxpipe organization.
* When you reinstall or rotate the Slack app, the bot leaves its channels.
Invite it again in each channel.
* All organizations of the deployment share the rate limits of the one Slack
app.
## Operator troubleshooting
### “Slack is not configured on this deployment”
The backend does not have both OAuth values. Confirm that `SLACK_CLIENT_ID` and
`SLACK_CLIENT_SECRET` are not empty in the backend runtime. Then redeploy or
restart the backend. A change to repository secrets or CI secrets alone does not
update a running service.
### OAuth returns `invalid_redirect_uri`
Register the exact callback:
```text
{AUTH_BASE_URL}/api/v1/connectors/slack/oauth/callback
```
Set `SLACK_REDIRECT_URI` only when you use a different public callback on
purpose. Its value must exactly match a redirect URL registered in Slack.
### Slack cannot verify the Events Request URL
* Confirm the public route is
`{AUTH_BASE_URL}/api/v1/webhook/slack`.
* Confirm `SLACK_SIGNING_SECRET` is Slack's **Signing Secret**, not the legacy
verification token.
* Confirm the public proxy forwards the raw request body and
`X-Slack-Signature` / `X-Slack-Request-Timestamp` headers unchanged.
* An unsigned test POST must return 401. A 404 means that the routing is wrong.
A 503 means that the backend has no signing secret.
### Mentions do not create commits
* Confirm Socket Mode is **off** and the bot Event Subscription includes
**`app_mention`**.
* Confirm that a Slack source with the status **Connected** lists the channel. If no
Workspace has the channel, the bot replies "Nothing captured: no Workspace
has this channel."
* Confirm the bot is a member of the channel. After an app reinstall, invite it
again.
* Mention the bot's **@handle** (Slack autocomplete), not only a display name
typed as plain text.
* Read the worker logs for the Slack mention capture job. The backend and the
worker must use the same database and stable encryption material
(`AUTH_SECRET` and, when set, `CONNECTION_SECRETS_ENCRYPTION_KEY`). The
GitHub App must still have contents write access to the Workspace repository.
Extra text after a mention needs `MODEL_PROVIDER*` (or Bedrock). A second
organization cannot install a workspace through OAuth when that workspace is already
connected.
### Capture commits but the repository never indexes
Capture writes to git through the GitHub App. Indexing starts only when this
deployment receives the GitHub **push** webhook for that repository. Confirm
that the GitHub App webhook URL points at **this** origin
(`{AUTH_BASE_URL}/api/v1/webhook/github/...`) and not at another environment.
Confirm that the worker and codesearch services run.
## Related pages
* [Configuration](/docs/self-hosting/configuration)
* [Slack connector (product)](/docs/connections/source-connectors/slack)
* [Confluence & Atlassian](/docs/self-hosting/confluence-atlassian)
* [GitHub App connector](/docs/self-hosting/configuration#github-app-connector)
* [Slack MCP server (external)](https://docs.slack.dev/ai/slack-mcp-server)
# AWS (https://docs.ctxpipe.ai/docs/self-hosting/deployment/aws)
`@ctxpipe/aws-cdk` is the AWS self-hosting path for ctxpipe. It gives you one high-level construct, `CtxPipe`. The construct provisions the required infrastructure and sets the runtime configuration for you.
Self-hosted ctxpipe on `@ctxpipe/aws-cdk` is **single-tenant**. Run one stack
per organization.
## Step-by-step deployment flow
### 1) Install `@ctxpipe/aws-cdk` in your CDK repo
Use an existing CDK repo, or create a new one first.
Then install:
```bash
pnpm add @ctxpipe/aws-cdk aws-cdk-lib constructs
```
### 2) Create or reuse a Route 53 hosted zone
You need a hosted zone for your custom domain because the construct configures
DNS and TLS from it.
* Reuse an existing public hosted zone if you already manage your domain in
Route 53.
* Or create a new hosted zone and delegate your domain or subdomain to it.
* Keep both values ready:
* `customDomain.domainName` (example: `app.example.com`)
* `customDomain.hostedZoneId` (example: `Z0123456789ABCDEF`)
### 3) Import `CtxPipe` and configure it
Add the construct to your stack. Follow the same pattern as
[`examples/aws-cdk-self-host`](https://github.com/ctxpipe-ai/ctxpipe/tree/main/examples/aws-cdk-self-host).
`modelProvider` is a union of two variants: **openai-like** (default) and **bedrock**. Both variants set the tiers through the same `models` prop (`fast` is required).
#### OpenAI-compatible provider
```ts
import * as cdk from "aws-cdk-lib";
import { CtxPipe } from "@ctxpipe/aws-cdk";
const app = new cdk.App();
const stack = new cdk.Stack(app, "CtxPipeStack");
const modelApiKey = app.node.tryGetContext("modelApiKey");
if (!modelApiKey) {
throw new Error('Missing CDK context "modelApiKey". Pass -c modelApiKey=...');
}
new CtxPipe(stack, "CtxPipe", {
orgSlug: "acme",
size: "medium",
customDomain: {
domainName: "app.example.com",
hostedZoneId: "Z0123456789ABCDEF",
},
modelProvider: {
baseUrl: "https://api.openai.com/v1",
apiKey: cdk.SecretValue.unsafePlainText(String(modelApiKey)),
models: {
fast: "gpt-4.1-mini",
embedding: "text-embedding-3-large",
// Optional: medium, high
// medium: "gpt-4.1",
// high: "gpt-4.1",
},
},
});
```
Set `models.embedding` for an OpenAI-compatible provider. Without it, the
backend uses `openai/text-embedding-3-large`, an OpenRouter model id that other
providers do not know.
#### Amazon Bedrock (task role)
For Bedrock, set `kind: "bedrock"` and the tier model IDs only. The construct grants the backend and worker ECS task roles `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream`. It injects `MODEL_PROVIDER=bedrock` and `MODEL_BEDROCK_AWS_REGION`. It does **not** create a `MODEL_PROVIDER_API_KEY` secret. The backend calls Bedrock Runtime with SigV4 credentials from the ECS task role.
Enable each model ID in the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock/) for your stack region before you use chat or embeddings.
Bedrock embeddings currently support only **Cohere** embed models. The backend uses the Cohere invoke format. When you omit `models.embedding`, the default is `cohere.embed-v4:0`. You can also use inference-profile IDs such as `global.cohere.embed-v4:0`. Non-Cohere Bedrock embed models (for example Titan) are not supported yet.
The construct attaches `bedrock:InvokeModel` and `bedrock:InvokeModelWithResponseStream` to the backend and worker task roles. An AWS Organizations **service control policy (SCP)** can override them with an **explicit deny**. The log text is typically `AccessDeniedException` … `explicit deny in a service control policy`. Allow your chat and Cohere embed foundation-model ARNs in the SCP. Denies for `global.cohere.embed-v4:0` often name `…/foundation-model/cohere.embed-v4:0`. This is a different problem from chat stream stalls such as `Bedrock Converse stream timed out … without receiving a chunk`. For those stalls, investigate `ConverseStream` in CloudTrail. A wider task-role policy does not help.
```ts
new CtxPipe(stack, "CtxPipe", {
orgSlug: "acme",
customDomain: {
domainName: "app.example.com",
hostedZoneId: "Z0123456789ABCDEF",
},
modelProvider: {
kind: "bedrock",
region: "us-east-1", // optional; defaults to stack region
models: {
fast: "anthropic.claude-sonnet-4-20250514-v1:0",
medium: "anthropic.claude-sonnet-4-20250514-v1:0",
high: "anthropic.claude-opus-4-20250514-v1:0",
embedding: "cohere.embed-v4:0", // optional; Cohere models only (e.g. global.cohere.embed-v4:0)
},
},
});
```
### 4) Run CDK deploy
**OpenAI-like:**
```bash
cdk synth
cdk deploy -c modelApiKey="$OPENAI_API_KEY"
```
For **Bedrock**, use the example app in
[`examples/aws-cdk-self-host`](https://github.com/ctxpipe-ai/ctxpipe/tree/main/examples/aws-cdk-self-host).
It always uses Bedrock. It reads `orgSlug`, `domainName`, `hostedZoneId`,
`size`, `stackName`, `modelFast`, `modelMedium`, `modelHigh`, and
`modelEmbedding` from the context:
```bash
cdk deploy \
-c modelFast="anthropic.claude-sonnet-4-20250514-v1:0" \
-c orgSlug=acme \
-c domainName=app.example.com \
-c hostedZoneId=Z0123456789ABCDEF
```
### 5) Wait for deployment and open your custom domain
After `cdk deploy` completes, open `https://`.
### 6) Complete onboarding with the same `orgSlug`
When you create the first organization in onboarding, use the **same** slug
as the `orgSlug` of `CtxPipe`. The construct uses the slug for the graph
database settings of the organization (`GRAPH_DB_URI_`). If the two
slugs are different, the graph of the organization does not work. The slug can
contain only lowercase letters, digits, and `-`.
### 7) Upgrade an existing stack
A new `@ctxpipe/aws-cdk` version on npm is the release unit. It pins the backend, worker, UI, codesearch, and migrate images to one git SHA. It also includes the Fargate size profiles of that release. After npm publishes the version:
```bash
pnpm update @ctxpipe/aws-cdk
cdk synth
cdk deploy
```
Before the first deploy of a new version, check the Aurora minor version. See
[Aurora PostgreSQL minor version](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#aurora-postgresql-minor-version).
`cdk deploy` runs Postgres migrations. Then it rolls the ECS services. Do not set image tags in `CtxPipe` props. Do not retag tasks to `:latest` yourself. The migrate task provisions `ctxpipe_app`. Then the stack writes the runtime `DATABASE_URL` secret. CDK operators do not set either URL by hand and do not run `psql`.
This way, the codesearch memory changes (4 / 8 / 12 GiB) and the ingest behavior arrive together. An update of only the container images, or only the task memory, leaves the stack split across releases.
The first upgrade to a version with the [chat sandbox host](#chat-sandbox-host)
adds it with no new required props. That deploy only adds resources and
replaces nothing that holds data. It waits until the host reports ready.
Then it rolls the backend and the worker.
The first upgrade to a version with the [knowledge store](#knowledge-store)
also adds it with no new required props, and replaces nothing that holds
data. The deploy waits until ClickHouse answers. Then the migrate task makes
the ClickHouse tables and users, and the deploy rolls the services.
To automate upgrades, let Renovate or Dependabot open a pull request for each
new `@ctxpipe/aws-cdk` version, and run `cdk synth` and `cdk deploy` from your
pipeline.
## Why each prop exists
### Required props
| Prop | Why it is required |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `orgSlug` | Self-hosted AWS CDK deployments are single-tenant. The slug identifies the one org for this stack and is used for org-scoped graph runtime values such as `GRAPH_DB_URI_`. |
| `customDomain` | ctxpipe needs one stable public HTTPS origin for browser traffic, auth, API routes, connector callbacks, and MCP. The construct uses this to configure DNS + TLS + ALB listeners. |
| `modelProvider` | Backend and worker need model IDs and either an OpenAI-compatible endpoint + API key, or Bedrock via ECS task role. See union below. |
### Optional props
| Prop | Behavior |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `size` | Capacity profile for single-tenant deployments. Valid values: `small`, `medium`, `large`. Defaults to `small` when omitted. |
| `otel` | Optional OTLP export. Omit it and the tasks export nothing. The construct does not deploy a collector or ClickStack. |
| `sandboxHost` | The [chat sandbox host](#chat-sandbox-host): `instanceType`, `dockerVolumeSizeGiB` (a whole number, at least 20), `agentVaultNetworkAllowlist` for a [private git server](#private-git-servers), and `enabled`. The host is on by default. `enabled: false` creates no host, and turns chat off. |
| `connectorSecrets` | Deployment-wide connector credentials. See [Optional connector secrets](#optional-connector-secrets). |
### `modelProvider` union
| Variant | `kind` | Required nested fields | Runtime auth |
| ----------------- | ------------------------ | ---------------------------------- | ----------------------------------------------------------------------- |
| OpenAI-compatible | omitted or `openai-like` | `baseUrl`, `apiKey`, `models.fast` | `MODEL_PROVIDER_API_KEY` in Secrets Manager |
| Amazon Bedrock | `bedrock` | `models.fast` | ECS task role → native Bedrock Runtime (SigV4); no model API key secret |
Optional on both variants: `models.medium`, `models.high`, and `models.embedding`.
Omitted tier IDs cascade: **medium** → **fast**, **high** → **medium**. The construct maps them to `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, `MODEL_HIGH_NAME`, and `MODEL_EMBEDDING_NAME`. See [Model configuration](/docs/self-hosting/models) for what each tier is used for.
For **bedrock**, `modelProviderSecret` / `modelProviderSecretArn` stack outputs are omitted.
### Upgrading from older `@ctxpipe/aws-cdk` versions
| Removed | Replacement |
| ---------------------------- | -------------------------------------- |
| `modelProvider.defaultModel` | `modelProvider.models.fast` (required) |
Replace `defaultModel: "gpt-4.1-mini"` with `models: { fast: "gpt-4.1-mini" }`.
### Required nested fields (openai-like)
| Field | Why it is required |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `customDomain.domainName` | Becomes the public app origin (`AUTH_BASE_URL` / `CTXPIPE_PUBLIC_APP_URL`) served over HTTPS. This is the URL users visit for sign-in and onboarding. |
| `customDomain.hostedZoneId` | Used to create/validate Route 53 records for ACM certificate validation, ALB alias records, and SES domain identity/DKIM records. |
| `modelProvider.baseUrl` | Endpoint for OpenAI-compatible model APIs used by backend and worker. |
| `modelProvider.apiKey` | Secret used to authenticate model API requests at runtime. |
| `modelProvider.models.fast` | Model ID for the fast tier (`MODEL_FAST_NAME`). |
### Required nested fields (bedrock)
| Field | Why it is required |
| --------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `customDomain.domainName` | Same as openai-like — public HTTPS origin. |
| `customDomain.hostedZoneId` | Same as openai-like — DNS/TLS/SES records. |
| `modelProvider.models.fast` | Bedrock model ID for the fast tier (`MODEL_FAST_NAME`). Enable it in the Bedrock console for the stack region. |
### Optional connector secrets
Pass deployment-wide connector credentials through `connectorSecrets`. Each
value is a `cdk.SecretValue`. The construct stores them in Secrets Manager and
gives them to the ECS tasks that need them. Most connectors can also take their
credentials in the setup wizard of the product, so these are optional.
| Connector | Keys |
| ---------- | -------------------------------------------------------------------------------------------------------- |
| GitHub App | `githubAppId`, `githubPrivateKey`, `githubWebhookSecret`, `githubClientId`, `githubClientSecret` |
| Atlassian | `atlassianClientId`, `atlassianClientSecret` |
| Slack | `slackClientId`, `slackClientSecret`, `slackSigningSecret` (set all three) |
| Linear | `linearClientId`, `linearClientSecret`, `linearRedirectUri`, `linearWebhookSecret` |
| Notion | `notionClientId`, `notionClientSecret`, `notionWebhookSecret` |
| PagerDuty | `pagerdutyClientId`, `pagerdutyClientSecret`, `pagerdutyRedirectUri`. There is no shared webhook secret. |
See
[Self-host Slack](/docs/self-hosting/slack) and
[Self-host PagerDuty](/docs/self-hosting/pagerduty) for callback and webhook
settings.
## Observability
| Prop | Env on backend, worker, UI, and codesearch |
| -------------------- | --------------------------------------------------- |
| `endpoint` | `OTEL_EXPORTER_OTLP_{TRACES,LOGS,METRICS}_ENDPOINT` |
| `headers` | `OTEL_EXPORTER_OTLP_HEADERS` (Secrets Manager) |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` |
[Configuration](/docs/self-hosting/configuration) describes browser telemetry and the signal URLs that the apps read. For the prop reference, see the [package README](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#observability).
## High-level architecture
`CtxPipe` provisions a single AWS stack with these layers:
| Layer | What is provisioned | Purpose |
| ------------------ | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------- |
| Network | VPC with public/private subnets, NAT egress, and an S3 gateway endpoint | Isolates internal services while allowing controlled outbound access. S3 traffic does not go through the NAT gateway. |
| Edge + ingress | Public ALB + ACM certificate + Route 53 records | Exposes one HTTPS origin and routes public traffic to backend. |
| Compute | ECS cluster + Fargate services (`backend`, `worker`, `ui`, `codesearch`) | Runs application APIs, background jobs, UI, and code indexing/search workloads. |
| Chat sandboxes | One EC2 Docker host in an Auto Scaling group, Docker volume, TLS secrets, CloudWatch alarms | Runs each Workspace conversation's agent in its own container. |
| Stateful data | Aurora PostgreSQL 16, Neptune cluster, EFS for `/data` | Persists relational state, graph data, and codesearch cache/indexes. |
| Knowledge store | One EC2 ClickHouse host in an Auto Scaling group, an EBS data volume, an S3 bucket for old parts | Holds the content of the [new knowledge store](/docs/self-hosting/configuration#knowledge-store). |
| Secrets + identity | Secrets Manager secrets, generated `AUTH_SECRET`, optional connector secrets | Stores runtime credentials and injects them into tasks securely. |
| Email delivery | SES domain identity, DKIM, SMTP credentials | Enables transactional email from backend (`ctxpipe-noreply@`). |
| Deploy safety | Migration custom resource (one-off ECS task) + ECS deployment rollback controls | Runs migrations during deploy and prevents partial/broken rollouts. |
Only the backend is public. UI and codesearch stay internal and are reached
through backend routing.
## Knowledge store
`CtxPipe` deploys ClickHouse 26.10 for the
[new knowledge store](/docs/self-hosting/configuration#knowledge-store). The
new knowledge store is off for every organization. While it is off, Neptune
answers, and the backend and the worker do not use ClickHouse.
### What it sets up
| Area | What the construct does |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Host | One Graviton EC2 instance in an Auto Scaling group of one, in the first private subnet. ClickHouse runs in Docker on the host network. The image is the same pinned image as in Docker Compose. |
| Data | An encrypted gp3 EBS volume holds the new parts and the cache of the old parts. The volume is not part of the instance: each new host attaches it again. When you delete the stack, CloudFormation keeps a snapshot of the volume. |
| Old parts | The knowledge tables move a part to the volume `cold` when its newest row is 30 days old. On AWS, `cold` is an S3 bucket of the stack behind a cache on the EBS volume (a fifth of the volume). The bucket blocks public access, is encrypted, and stays when you delete the stack. When the volume has less than 20% free space, ClickHouse also moves its largest parts to the bucket. The VPC has an S3 gateway endpoint, so this traffic does not go through the NAT gateway. |
| Network | Only the backend, the worker, and the migrate task reach ClickHouse, on port 8123 (HTTP), at `clickhouse.ctxpipe.local`. The host registers this Cloud Map name when it boots. There is no public listener and no SSH key. |
| Users | The stack generates three passwords in Secrets Manager. The migrate task gets the admin user and the two other passwords, and makes the users `ctxpipe_reader` and `ctxpipe_writer`. The backend gets `ctxpipe_reader` (read only). The worker gets `ctxpipe_writer`. Each task gets its own `CLICKHOUSE_URL`. |
| Credentials | ClickHouse reads and writes the bucket with the instance role. The role reaches only the prefix `clickhouse/` of the bucket. |
| Logs | ClickHouse logs to CloudWatch Logs (14 days). Of the server's own log tables, only `system.query_log` and `system.error_log` stay, for 7 days. |
The output `KnowledgeStoreAutoScalingGroupName` names the group. Open a shell
on its instance with Session Manager for checks.
### Knowledge store alarm
| Alarm | Fires when | What to do |
| ------------ | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Status check | The ClickHouse host fails its EC2 status checks for 2 minutes | The Auto Scaling group replaces the host, and the new host attaches the same data volume. If the alarm stays on, look at the group's activity history and the ClickHouse log group. |
The alarm has no actions. Subscribe it to your own topic through
`ctxPipe.knowledgeStoreAlarms`, as for the
[chat sandbox host alarms](#alarms). The host has EC2 detailed monitoring,
because EC2 publishes the status checks by Auto Scaling group only with it.
### Sizes and cost
| `size` | Instance | vCPU / RAM | Data volume (gp3) | \~Cost/month |
| ----------------- | ------------ | ---------- | ----------------- | ------------ |
| `small` (default) | `t4g.large` | 2 / 8 GiB | 50 GB | \~$55 |
| `medium` | `r7g.large` | 2 / 16 GiB | 100 GB | \~$88 |
| `large` | `r7g.xlarge` | 4 / 32 GiB | 200 GB | \~$174 |
Costs are us-east-1 on-demand prices for the instance, the data volume, and a
16 GB root volume. The S3 storage of the old parts and the detailed monitoring
of the host (a few dollars a month, for the status check alarm) add to this. A move to a
larger `size` replaces the host and makes the volume larger. Do not move to a
smaller `size`: EBS cannot make a volume smaller, and the deploy fails.
### Recovery
ClickHouse holds only content that ctxpipe makes from the Workspace
repositories. A replacement host attaches the same volume, so it keeps the
data. If the volume is lost, follow
[Recover the knowledge store](/docs/self-hosting/configuration#recover-the-knowledge-store).
On AWS, the next `cdk deploy` that changes the migrate task runs the
migrations.
A single host is a single point of failure: while it is replaced, the new
knowledge store does not answer.
## Chat sandbox host
Workspace chat runs each conversation's agent (OpenCode) in its own Docker
container. `CtxPipe` creates one EC2 host for those containers by default. The
sandbox keeps the agent's commands away from the backend and your data stores.
To create no host, set `sandboxHost: { enabled: false }`. Without the host,
Workspace chat is off. The product answers chat requests with a "sandbox
unavailable" error (503). The rest of the product keeps working.
### Sizes and cost
The host mostly holds idle sandboxes, so it uses burstable Graviton (ARM)
instances. Memory is the limit. An active sandbox uses about 0.3–0.5 GiB.
Sandboxes stop after 5 minutes idle and resume on the next message. Stopped
sandboxes are removed 30 days after last use. An organization runs at most 50
sandboxes at once.
| `size` | Instance | vCPU / RAM | Docker volume (gp3) | \~Cost/month | Running sandboxes (est.) |
| ----------------- | ------------ | ---------- | ------------------- | ------------ | ------------------------ |
| `small` (default) | `t4g.medium` | 2 / 4 GiB | 30 GB | \~$28 | \~6 |
| `medium` | `t4g.large` | 2 / 8 GiB | 50 GB | \~$54 | \~14 |
| `large` | `t4g.xlarge` | 4 / 16 GiB | 100 GB | \~$107 | \~30 |
Costs are us-east-1 on-demand prices for the instance, the Docker volume, and
a 16 GB root volume. `t4g` instances run in unlimited mode, so sustained CPU
above baseline adds a small surcharge.
Override either value with `sandboxHost`:
```ts
import * as ec2 from "aws-cdk-lib/aws-ec2";
new CtxPipe(stack, "CtxPipe", {
// ...orgSlug, customDomain, modelProvider
sandboxHost: {
instanceType: ec2.InstanceType.of(ec2.InstanceClass.T4G, ec2.InstanceSize.XLARGE),
dockerVolumeSizeGiB: 150,
},
});
```
The instance type must be Graviton (arm64). Synth rejects other architectures.
### Private git servers
Ingestion can read a plain git repository on a git server inside your network
(see [Plain git repositories](/docs/workspaces/linked-repositories#plain-git-repositories)).
The ingestion sandbox fetches it through Agent Vault on the sandbox host.
Agent Vault connects only to the private addresses in its allowlist. By
default, the allowlist holds only the backend subnets. Add the single host
address of your git server with `sandboxHost.agentVaultNetworkAllowlist`:
```ts
new CtxPipe(stack, "CtxPipe", {
// ...orgSlug, customDomain, modelProvider
sandboxHost: {
agentVaultNetworkAllowlist: ["10.20.0.15/32"],
},
});
```
Chat-sandbox vaults forward each host that no rule matches. An address or CIDR in this list therefore lets every chat agent reach every port and path at that address. Add only the single host address of the git server. Never add a whole subnet.
The stack adds each entry after the backend subnets. Each entry must be an
IPv4 or IPv6 address or CIDR. Synth rejects an entry with a cloud metadata or
link-local address. The sandbox host must also have a route to that address,
for example VPC peering or a transit gateway. The git server must accept
connections from the host. A change to the list replaces the host, as each
change to its boot script does.
### What it sets up
| Piece | Behavior |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Instance | Amazon Linux 2023 in a single-instance Auto Scaling group in the private subnets. EC2 health checks replace a failed host. `cdk deploy` waits until the host reports ready. New AMIs apply when the host is replaced, not on every deploy. |
| Docker volume | gp3 volume at `/var/lib/docker`. It belongs to the instance, so a replaced host starts empty. Conversations keep their work in their git branch and continue on the new host. |
| Docker API | Port 2376 with mutual TLS at `sandbox-host.ctxpipe.local`, a Cloud Map name the host registers when it boots. The host creates the certificates on first boot and stores them in Secrets Manager; a replacement host reuses them. The CA key is not kept. |
| Network | Backend and worker reach the Docker API and the Agent Vault API (port 14321). Only the backend reaches the sandboxes' agent ports. `DOCKER-USER` rules let sandboxes reach only the Agent Vault proxy (port 14322); a direct connection, a non-HTTP protocol, and DNS are rejected. Docker forwards the proxy port without NAT, so Agent Vault sees the address of each sandbox: failed proxy logins from one sandbox do not block the proxy for the other sandboxes. Through the proxy, sandboxes reach Git hosts, the internet, and the backend (model proxy and per-run tool bridges), but not Postgres, Neptune, ClickHouse, EFS, codesearch, the UI, or the worker. This is the [chat sandbox network policy](/docs/self-hosting/architecture#chat-sandbox-network-policy) shared with Docker Compose. |
| Credentials | Agent Vault (`infisical/agent-vault:latest`) runs on the host on its own Docker bridge. Sandboxes hold no credential: Agent Vault adds the GitHub token and the backend credentials to each request. Agent Vault may dial the backend's subnets, and the entries of `sandboxHost.agentVaultNetworkAllowlist`. The backend subnets are the stack's only private subnets, so they also hold RDS, Neptune, the ClickHouse host, EFS, and the UI and codesearch tasks; only security groups keep the proxy out of them. If you change security groups, never admit the sandbox host's group to those services. The stack generates its master password and owner password in Secrets Manager; the host reads the master password, and backend and worker get the owner password. If Agent Vault does not answer, chat fails closed. A replaced host starts an empty Agent Vault, and the backend registers again on first use. |
| Isolation | Each agent port needs a per-conversation password. Agent ports are reachable inside the VPC, so the host also blocks sandboxes from each other, from the host itself, and from instance metadata. |
| Limits | All sandbox containers share one systemd slice capped at 85% of host memory. Container logs rotate at 3 × 10 MB. |
| Chat image | `ghcr.io/ctxpipe-ai/chat-sandbox`, published for arm64 and amd64 with the same release tag as the service images. The backend pulls it through the host's Docker daemon on the first chat after a release, so releases never replace the host. |
| Backend and worker | Get `SANDBOX_PROVIDER=docker`, `SANDBOX_CHAT_IMAGE`, `DOCKER_HOST=tcp://sandbox-host.ctxpipe.local:2376`, `DOCKER_TLS_VERIFY=1`, and `DOCKER_CERT_PATH`. A short init container in each task writes the client certificates there as files. They also get `AGENT_VAULT_ADDR` and the `AGENT_VAULT_OWNER_PASSWORD` secret. Sandboxes call the backend back at its task's VPC DNS name (`ip-….compute.internal`), so they reach the replica that runs their turn. The stack sets `SANDBOX_CALLBACK_DNS_SUFFIX` for this. The VPC must have DNS host names and DNS resolution on; the stack's VPC has both. |
### Alarms
| Alarm | Fires when | What to do |
| ----------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Docker disk | `/var/lib/docker` is 80% full or more for 10 minutes | Check `docker system df` on the host (see [Operations](/docs/self-hosting/operations#chat-sandbox-host-aws)). Raise `sandboxHost.dockerVolumeSizeGiB` if the volume fills with live sandboxes. |
| Memory | Host memory is 85% used or more for 15 minutes | Sandboxes are near their cap. Move to a larger `size` or `sandboxHost.instanceType`. |
The alarms have no actions. Subscribe them to your own topic through the
construct:
```ts
import * as actions from "aws-cdk-lib/aws-cloudwatch-actions";
const ctxPipe = new CtxPipe(stack, "CtxPipe", { /* ... */ });
for (const alarm of ctxPipe.sandboxHostAlarms) {
alarm.addAlarmAction(new actions.SnsAction(opsTopic));
}
```
The backend and worker images refuse `SANDBOX_PROVIDER=unsandboxed`. An
agent without a sandbox would run with the backend's credentials. `CtxPipe`
configures a Docker host for chat sandboxes.
## What this gives you by default
* `GRAPH_DB_PROVIDER=neptune` and graph URI wiring from Neptune endpoints
* Internal service routing (`UI_PROXY_URL`, `CODESEARCH_URL`) so only backend is public
* Generated `AUTH_SECRET` and managed `DATABASE_URL` (runtime `ctxpipe_app`) / SMTP secrets
* A `CLICKHOUSE_URL` for each of backend, worker, and migrate, each with its own ClickHouse user
* Public app URL output and secret ARN outputs for operations
## How to choose `size`
Use `size` to tune compute and database capacity for your single-tenant workload:
* `small` (default): for pilot and small-team deployments with moderate repository churn and cost sensitivity. Aurora and Neptune both use `db.t4g.medium`. This is the minimum burstable size that AWS supports for these engines (for dev and test). Codesearch is 4 GiB (ingest peak, not idle RSS).
* `medium`: for sustained ingestion and reindex workloads where the worker and codesearch need more headroom. Codesearch is 8 GiB. Aurora uses `db.t4g.large`; Neptune uses memory-optimized `db.r6g.large`.
* `large`: for high ingestion and lower latency. It has bigger tasks, scaled backend and worker replicas, and `db.r6g.xlarge` for both Aurora and Neptune. Codesearch is 12 GiB and still **one task**. The two workers each get `OPENWORKFLOW_CONCURRENCY=8`, so the cluster cannot overload codesearch. Extra workers help extract and connectors. They do not help clone or SCIP parallelism.
See the [package README sizing table](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md#sizing-profiles) for the full preset matrix.
Do **not** raise the desired count of codesearch. Ingest throughput grows with codesearch memory (4 / 8 / 12 GiB) and the environment variables that the construct injects (`OPENWORKFLOW_CONCURRENCY`, `CODESEARCH_INDEXER_CONCURRENCY`, `CODESEARCH_INDEX_PIPELINE_CONCURRENCY`).
Signals that you should move up a size:
* ingestion jobs queueing up or taking too long,
* frequent codesearch reindex delays,
* higher p95 response latency during active sync/index windows.
`@ctxpipe/aws-cdk` keeps services in private subnets for every size. You still need NAT, because the backend, worker, and codesearch need outbound access to model providers and to the GitHub, Atlassian, and Linear APIs.
## Related references
* [`packages/aws-cdk/README.md`](https://github.com/ctxpipe-ai/ctxpipe/blob/main/packages/aws-cdk/README.md)
* [`examples/aws-cdk-self-host`](https://github.com/ctxpipe-ai/ctxpipe/tree/main/examples/aws-cdk-self-host)
# Docker (https://docs.ctxpipe.ai/docs/self-hosting/deployment/docker)
Docker Compose is the reference self-hosted path. Use it for an evaluation, for
production-style tests, and for a small internal deployment on one host.
## Start the stack
You need Docker with Compose, Node.js, and pnpm. Run these commands from a clone
of the repository:
```bash
git clone https://github.com/ctxpipe-ai/ctxpipe.git
cd ctxpipe
cp docker-compose.env.example .env
```
Set at least these values in `.env`:
```bash
AUTH_SECRET=<32-plus-character-secret>
AUTH_BASE_URL=https://ctx.example.com
CTXPIPE_PUBLIC_APP_URL=https://ctx.example.com
MODEL_PROVIDER=openrouter
MODEL_PROVIDER_API_KEY=
```
Then start the deploy profile:
```bash
pnpm start
```
`pnpm start` runs `docker compose --profile deploy up -d`. It builds the
images only when they do not exist.
For a local evaluation, `AUTH_BASE_URL` and `CTXPIPE_PUBLIC_APP_URL` can be
`http://localhost:3000`. For team use, put a TLS proxy in front of the backend
and use the public HTTPS origin. The UI image contains `CTXPIPE_PUBLIC_APP_URL`
at build time. After you change it, build the images again with
`docker compose --profile deploy up -d --build`.
## What the deploy profile starts
| Service | Purpose |
| --------------------- | -------------------------------------------------------------------------------------- |
| `migrate` | Runs the database migrations as the Postgres owner, then gives access to `ctxpipe_app` |
| `backend` | The public API, auth, MCP, callbacks, and the UI proxy. Published on host port `3000`. |
| `ui` | The web app, served through the backend |
| `worker` | The background jobs for ingestion and connectors |
| `codesearch` | The repository clone cache and the Zoekt indexes. Keep one replica. |
| `postgres` | The application database |
| `falkordb` | The default graph database |
| `clickhouse` | The knowledge content |
| `dind` | The Docker daemon for chat and ingestion sandboxes |
| `chat-sandbox-image` | One-shot: builds the sandbox image inside `dind` |
| `agent-vault` | Adds credentials to sandbox requests. It is the only address that sandboxes can reach. |
| `agent-vault-secrets` | One-shot: generates the Agent Vault passwords |
The backend is the only service on a public host port (`3000`). Compose binds
`codesearch` (`3001`) and `ui` (`3002`) to `127.0.0.1` only. Postgres,
FalkorDB, and ClickHouse have no host port.
Compose gives each service its own ClickHouse user in `CLICKHOUSE_URL`:
| Service | ClickHouse user | Password in `.env` | Access |
| --------- | ----------------- | ---------------------------- | ------------------------------------------------------------------------------------------------- |
| `migrate` | `CLICKHOUSE_USER` | `CLICKHOUSE_PASSWORD` | Admin. It makes the tables and the two other users. |
| `backend` | `ctxpipe_reader` | `CLICKHOUSE_READER_PASSWORD` | Reads the knowledge tables only |
| `worker` | `ctxpipe_writer` | `CLICKHOUSE_WRITER_PASSWORD` | Reads, inserts, and deletes rows in the knowledge tables. It cannot manage users or row policies. |
Set `CLICKHOUSE_READER_PASSWORD` and `CLICKHOUSE_WRITER_PASSWORD` before the
first start. They have no default. Without them, `migrate` does not make the
two users, and the backend and the worker cannot connect to ClickHouse.
The default admin user and password are `ctxpipe`. Set your own
`CLICKHOUSE_PASSWORD` too. `migrate` refuses the reader and writer passwords
when the admin password is `ctxpipe`, or when two of the three passwords are
the same.
Use only URL-safe characters in each password. `CLICKHOUSE_DB` sets the
database (default `ctxpipe`). To change a reader or writer password, change it
in `.env` and run `migrate` again.
## Settings that need an override file
The deploy profile passes only a fixed list of variables from `.env` to each
service. The list has the auth, model, OpenTelemetry, and sandbox settings, and
the Slack, Linear, Notion, and PagerDuty connector secrets. Other values in `.env` do not
reach the services. Add them to the `environment` of `backend` and `worker` in
a `docker-compose.override.yml`. This applies to:
* email: `SMTP_CONNECTION_URL` and `EMAIL_FROM_ADDRESS`. Without them, ctxpipe
sends no verification, password reset, or invitation email;
* social sign-in: `GITHUB_CLIENT_ID`, `GOOGLE_*`, and `MICROSOFT_*`;
* Atlassian OAuth: `ATLASSIAN_CLIENT_ID` and `ATLASSIAN_CLIENT_SECRET`;
* an external graph database: `GRAPH_DB_URI`, `GRAPH_DB_PROVIDER`,
`GRAPH_DB_USER`, and `GRAPH_DB_PASSWORD`.
```yaml
services:
backend:
environment:
SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL}
EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS}
worker:
environment:
SMTP_CONNECTION_URL: ${SMTP_CONNECTION_URL}
EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS}
```
An S3 disk for the old ClickHouse parts also needs an override file: mount
your storage file on the `clickhouse` service. See
[Keep old parts on S3](/docs/self-hosting/configuration#keep-old-parts-on-s3).
The in-product GitHub wizard stores the GitHub App credentials in the database.
The GitHub App therefore needs no variable.
## Persistent state
The deploy profile mounts no named volume for `postgres` or `falkordb`:
* Postgres keeps its data in an anonymous Docker volume. `docker compose down
-v` deletes it. After `docker compose down`, the new container does not use
the old volume.
* The FalkorDB image declares no volume, so FalkorDB keeps its data in the
writable layer of the container. Each time Compose creates the container
again, you lose the graph data. This happens, for example, after `down`, an
image update, or a configuration change.
Before people depend on the deployment, mount named volumes in
`docker-compose.override.yml`:
```yaml
services:
postgres:
volumes:
- postgres_data:/var/lib/postgresql/data
falkordb:
volumes:
- falkordb_data:/var/lib/falkordb/data
volumes:
postgres_data:
falkordb_data:
```
Do this before the first start. On a running deployment, first back up the
data. Then restore it into the new volumes.
| State | Where it is | Back it up |
| ------------------------------ | --------------------------------------------------------------------- | -------------------------------------------------------------- |
| Postgres data | An anonymous volume of the `postgres` container, or your named volume | Yes |
| Graph data | The `falkordb` container, or your named volume | Yes, or rebuild it from the Workspace repositories |
| Knowledge content | `clickhouse_data` | Yes, or rebuild it from the Workspace repositories |
| Repository clone cache | `repo_cache` | No, codesearch clones again |
| Zoekt indexes | `zoekt_index_deploy` | No, codesearch indexes again |
| Agent Vault passwords and data | `agent_vault_master`, `agent_vault_owner`, `agent_vault_data` | Keep them. New passwords do not match an existing Agent Vault. |
| Sandboxes and TLS certificates | `sandbox_docker`, `sandbox_ca`, `sandbox_client_certs` | No |
| Secrets | `.env`, or your secret store | Yes |
To back up Postgres, run `docker compose exec postgres pg_dump -U ctxpipe ctxpipe > backup.sql`.
### Database passwords
Compose sets two Postgres URLs for you. `migrate` connects as `POSTGRES_USER`
(default `ctxpipe`). The other services connect as `ctxpipe_app`. Both use
`POSTGRES_PASSWORD`. To change the password:
1. Change the owner password in Postgres:
`docker compose exec postgres psql -U ctxpipe -c "ALTER ROLE ctxpipe PASSWORD ''"`.
Postgres reads `POSTGRES_PASSWORD` only when it creates the volume.
2. Set the same value as `POSTGRES_PASSWORD` in `.env`.
3. Run `docker compose --profile deploy up -d`. `migrate` sets the new
`ctxpipe_app` password, and the services restart with it.
## Chat sandboxes
Each conversation and each ingestion runs in its own container in `dind`, a
Docker-in-Docker daemon. It needs no settings.
* `dind` is the only privileged container. The sandboxes are not privileged,
and the backend never mounts the Docker socket of the host.
* The Docker API listens on port `2376` inside Compose, with mutual TLS. It is
not published on the host.
* Sandboxes hold no credential. They can reach only Agent Vault. See the
[sandbox network policy](/docs/self-hosting/architecture#chat-sandbox-network-policy).
* `dind`, `agent-vault`, `backend`, and `worker` share the `sandbox` network,
`172.30.99.0/24`. If it overlaps a network of your host, change each
`172.30.99.x` address in `docker-compose.yml`.
* `chat-sandbox-image` builds the sandbox image on each `pnpm start`. Until the
image exists, chat answers `503`, and the rest of the app works.
* On a cgroup v2 host, all sandboxes share one limit of 8192 processes. Without
cgroup v2, `dind` logs a warning and applies no limit.
* Container logs rotate at 3 files of 10 MB.
* If you remove `sandbox_ca`, `dind` makes new certificates. Then restart
`backend` and `worker`.
To limit the memory of all sandboxes, limit `dind` in
`docker-compose.override.yml`. The sandboxes together get at most 85% of it:
```yaml
services:
dind:
mem_limit: 8g
```
Each process that has the files in `sandbox_client_certs` can start
privileged containers on `dind`. That access is root on the Docker host. Do not
mount that volume in another service. Do not copy it off the host.
If `dind` stops, chat answers `503` until it is back. It never runs an agent
without a sandbox. The backend and worker images refuse
`SANDBOX_PROVIDER=unsandboxed`, because an agent without a sandbox has the
credentials of the backend. If your host cannot run privileged containers, chat
is not available.
To read a plain git repository on a private git server, see
[Plain git repositories](/docs/self-hosting/configuration#plain-git-repositories).
## Check the deployment
* `docker compose --profile deploy ps` shows healthy services and a healthy
`dind`.
* `curl https:///.status` returns `ok`.
* You can sign in through the public origin.
* A linked repository reaches **Indexed**.
* Workspace chat answers, and `docker compose exec dind docker ps` shows its
sandbox.
* An MCP client reaches `{AUTH_BASE_URL}/mcp`.
See [Operations](/docs/self-hosting/operations#chat-sandboxes-docker-compose)
for the sandbox disk and health checks.
# Buildkite connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/buildkite)
The **Buildkite connector** writes the latest builds of selected pipelines into
a [Workspace's](/docs/workspaces) GitHub repository. ctxpipe then indexes them
like other knowledge files. An agent can see which builds failed, which jobs
failed, and the commit and pull request of each build. A source file in the
Workspace holds the pipeline scope. A pull request adds it, so a person reviews
the scope before content syncs.
ctxpipe only reads from Buildkite. It does not start, retry, or cancel builds.
## What ctxpipe syncs
You select pipelines for each Workspace. For each pipeline, ctxpipe writes:
* `pipeline.md`: the name, repository, default branch, and link.
* One file for each of the latest 25 finished builds: the state, branch,
commit, pull request, the person who created the build, and the jobs that
failed, with a link to each. A soft-failed job does not fail the build, so
it is not listed.
A finished build is passed, failed, canceled, or blocked. A blocked build
waits at a block step. When someone unblocks it, the build continues and is no
longer finished. The next sync deletes its file. The file comes back when the
build finishes.
ctxpipe does not read build logs, environment variables, meta-data, or the
commands of steps. These can contain secrets.
When the repository of a pipeline is on GitHub, a build file links to its
commit and pull request. The Workspace graph then joins the build to them.
## Prerequisites
* A Buildkite user that can read the pipelines of the organization.
* A [Workspace](/docs/workspaces) whose repository the ctxpipe GitHub App can
write to. The repository must use format 2 or higher. If it does not, the
dialog shows **Needs the format upgrade first.** next to the Workspace. See
[Upgrades](/docs/workspaces/settings#upgrades).
* Permission to merge the pull request that adds the source.
* Optional: permission to add a webhook in the Buildkite organization
settings.
## Where the scope lives
The connection holds the tokens only. The scope is in a source file in each
Workspace, at `sources/buildkite/.md`. The front matter holds the
pipeline slugs. The body is an optional description of what the source is for.
```markdown
---
provider: buildkite
connection: con_example
pipelines: [web-app, deploy-production]
---
```
The scope goes live after the merge. See
[Workspace settings](/docs/workspaces/settings#sources) for draft scope and
live scope. Each Workspace has its own scope. One source holds at most 50
pipelines.
## Setup flow
### Create an API access token
In Buildkite, open
[API Access Tokens](https://buildkite.com/user/api-access-tokens/new).
Select only your organization. Select only these REST scopes:
`read_pipelines` and `read_builds`. Do not select **Enable GraphQL API
access**.
That option gives the token full access, including write access. ctxpipe
refuses a token that has any other scope or GraphQL API access, because
the connector only needs these two scopes.
### Connect Buildkite
Open **Connectors**, select **Add connection**, then select **Connect** on
the **Buildkite** row. Type the organization slug (the name in your Buildkite URL) and paste
the token. ctxpipe checks the token with Buildkite before it saves it. ctxpipe
keeps the token encrypted and never shows it again.
### Add the webhook
The webhook tells ctxpipe when a build finishes. In Buildkite, open
**Settings → Notification Services**, and add a **Webhook**:
1. **Webhook URL:** paste the URL that the setup shows.
2. **Token:** select the signature option (`X-Buildkite-Signature`).
ctxpipe refuses a plain token.
3. **Events:** clear every event except `build.finished`.
4. **Pipelines:** keep all pipelines, or select the pipelines that your
Workspaces sync. Then save the webhook.
5. Copy the token of the webhook, and paste it into the setup.
To add the webhook later, close the setup. The connector card shows
**Add the webhook**, and **Add to a Workspace** under **Used in**. Until
then, ctxpipe syncs once a day and when you select **Sync now**.
### Add to Workspaces
The next step is **Add Buildkite to Workspaces**.
1. Select the Workspaces that sync from Buildkite.
2. Select at least one pipeline. Type in the filter to find a pipeline by
its name or slug.
3. If you select more than one Workspace, choose **Same scope for each
Workspace** or **A different scope for each Workspace**.
4. Choose **Add to Workspaces**.
If the organization has no Workspace yet, the step shows **Create a
Workspace**. Create the Workspace, then return to **Connectors** and
choose **Add to a Workspace** on the Buildkite card. The connection stays
saved.
You can add the connection to a Workspace later. Expand the connector on
**Connectors**. When no Workspace uses the connection, choose **Add to a
Workspace**. Otherwise, choose **Manage**.
### Merge the pull requests
ctxpipe opens one pull request for each Workspace. Each pull request adds
`sources/buildkite/.md`. The dialog shows **Merge the pull request**
(or **Merge the pull requests**) with a link to each one.
1. Review the selected pipelines and the repository audience.
2. Merge each pull request.
After each merge, the dialog shows **Pull request merged**. The source does
not sync before the merge.
### Wait for the first sync
After GitHub reports the merge, ctxpipe reads the source file and writes the
files of each pipeline. The Workspace indexes the new files after that. See
[Indexing](/docs/workspaces/indexing) for the lifecycle.
## Change what syncs
Edit, rename, or remove a source in the **Settings** of the Workspace. See
[Workspace settings](/docs/workspaces/settings#sources). When you remove a
pipeline from the scope, ctxpipe deletes the files of that pipeline in a sync
commit.
## Files in the repository
```text
buildkite/pipelines//pipeline.md
buildkite/pipelines//builds/.md
```
Do not edit these files. ctxpipe writes them again on each sync. When a build is
no longer among the latest 25, ctxpipe deletes its file in the same commit.
## How builds stay current
* **Webhook:** a signed `build.finished` event starts a sync of the connection
at the end of a 10-minute window. More events in the same window start no
more syncs, so a busy pipeline gives at most one commit for each Workspace in
each window.
* **Daily sync:** ctxpipe syncs each connection once a day.
* **Sync now:** an organization owner or admin can select **Sync now** in the
menu of the connector card.
## Verify the complete flow
1. Connect Buildkite and confirm that the organization slug appears on
**Connectors**.
2. Add the webhook, and confirm that the card does not show **Add the
webhook**.
3. Choose **Add to Workspaces**, select a Workspace, and select at least one
pipeline.
4. Merge the pull request that adds `sources/buildkite/.md`.
5. Confirm that the Workspace repository has files under
`buildkite/pipelines/`.
6. Finish a build in a selected pipeline. Confirm that its build file appears
at the end of the 10-minute window.
7. Confirm that the Workspace index is ready in its **Settings**.
Steps 4 and 5 test the GitHub push webhook and the first sync. Step 6 tests
the Buildkite webhook delivery.
## Troubleshooting
### Buildkite refuses the token
The setup shows the reason. Make sure that the token has the two read
scopes and no GraphQL API access. Make sure that the organization of the token
is the organization slug that you typed. If the setup says that ctxpipe cannot
make sure that the token has no GraphQL API access, ask the operator to check
that the backend can reach `graphql.buildkite.com`.
### Add to a Workspace shows an error
The dialog shows a message about the format, the write access, or your role.
See [Troubleshooting](/docs/connections#troubleshooting).
### No pull request appears
Confirm that the GitHub App has write access to the Workspace repository. If
the dialog reports an error, close it and try again.
### The source shows a status other than OK
Read the status and the reason of the row in **Settings**. See
[Source status](/docs/workspaces/settings#source-status).
### Content sync failed
Choose **Sync now** in the menu of the connector card. If it fails again, ask
the operator to check the worker logs, GitHub write access, and outbound access
to `api.buildkite.com`.
### The card shows “Needs reconnect”
Buildkite refused the saved token, for example because someone deleted it.
The card says "Buildkite does not accept the saved token. Reconnect with a new
token." The source shows the reason "Buildkite does not accept the API access
token of this connection any more. Reconnect Buildkite on the Connectors page."
ctxpipe keeps the files and stops the sync. Select **Reconnect Buildkite** on
the card and paste a new token.
### A source shows “Out of access”
A selected pipeline was deleted or renamed, or the token cannot read it. ctxpipe
tries the pipeline in three sync cycles and keeps its files. Then it marks the
source. Remove the pipeline from the source, or select it again by its new
name.
### Builds do not arrive after they finish
Make sure that the webhook uses the signature option, sends `build.finished`,
and includes the pipeline. A build arrives at the end of the 10-minute window
of its event. The webhook settings in Buildkite show the response of each
delivery. A `401` response means that the token in ctxpipe is not the token of
the webhook. Expand the connector card, select **Replace the webhook token**,
and paste the token again.
### Remove the connector
Choose **Remove connector** in the menu of the connector card. See
[How content stays current](/docs/connections#how-content-stays-current) for
what stays in Git. This deletes the saved tokens. The Workspace sources that use the connector
stop syncing. Delete the webhook in Buildkite too, and the API access
token if nothing else uses it.
# Confluence connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/confluence)
The **Confluence connector** mirrors the Confluence spaces and pages that you
select into a [Workspace](/docs/workspaces) repository. ctxpipe does not read
the whole site. You choose what agents can use.
ctxpipe connects to the Confluence site through Atlassian OAuth and the ctxpipe
Forge app. When pages or spaces change, the Forge app tells ctxpipe, and
ctxpipe updates the mirror.
On a self-hosted deployment, the operator provides the Atlassian OAuth app and
the Forge setup first. See
[Self-host Confluence & Atlassian](/docs/self-hosting/confluence-atlassian).
## Data flow
## Before you start
You need:
* permission to authorize Atlassian OAuth for the Confluence site;
* permission to install the ctxpipe Forge app on the site, or an Atlassian
admin who can install it.
Open `admin.atlassian.com`, choose the Confluence site, and check whether you
can open **Connected apps** and install apps. Atlassian requires organization
admin or site admin access for this. If you cannot, ask an Atlassian admin to
install the Forge app. See
[Atlassian Connected apps](https://support.atlassian.com/organization-administration/docs/managing-an-installed-app/).
## Connect Confluence
### Start from Connectors
Open **Connectors**, choose **Add connection**, then **Connect** on the
**Confluence** row. The setup wizard tracks each task until the connector is
ready.
### Link your Atlassian account
Link your Atlassian account with OAuth. This tells ctxpipe which Atlassian
cloud site you connect. If the account is already linked to another ctxpipe
organization, the wizard asks you to confirm or cancel.
### Install the Forge app
Install the ctxpipe Forge app on the same Atlassian site. The Forge app lets
ctxpipe read the content that you select, and tells ctxpipe about changes.
### Add to Workspaces
Select the Workspaces, then select the spaces and pages in the tree. For
each space, select all pages or only some. Choose the scope with care:
* start with the spaces that your engineering team uses;
* leave out old or sensitive areas that agents must not rely on;
* choose the authoritative docs, not duplicates or abandoned pages.
Choose **Add to Workspaces**, then merge the pull request that adds
`sources/confluence/.md`. See
[Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace).
## Source file
For each space, `pages` is `all` or a list of page ids:
```markdown
---
provider: confluence
connection: con_example
spaces:
- key: ARCH
pages: all
- key: OPS
pages: ["12345", "67890"]
---
```
## Files in the repository
Mirrored pages are under `confluence//`, in the page tree of the
space. A page is `--.md`, or `--/index.md` when it
has child pages. Attachments and embedded media are under
`confluence//_assets//`. Each asset can be up to 25 MiB, and
each page up to 100 MiB in total. A file that is too large or not safe stays a
link to Confluence. ctxpipe never writes Atlassian download URLs to git.
For problems, see [Troubleshooting](/docs/connections#troubleshooting).
# GitHub connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/github)
GitHub is the first connector to set up. The ctxpipe GitHub App gives ctxpipe
access to your [Workspace](/docs/workspaces) repositories and to the code
repositories that you link. The other connectors, including Buildkite, write
their content into a Workspace repository on GitHub, so they need it too.
Hosted ctxpipe supplies the GitHub App. A self-hosted deployment creates and
owns its own App. The operator completes
[Self-host GitHub](/docs/self-hosting/github) first.
## Connect GitHub
### Start the connection
Choose **Connect GitHub** in the **Get started** guide on Home, or open
**Connectors** and choose GitHub.
### Install the App
Install the ctxpipe GitHub App on the GitHub account or organization that
owns your repositories. If you cannot install apps there, GitHub sends a
request to an organization owner.
After the install, GitHub can ask you to authorize the ctxpipe App. This
shows ctxpipe that your GitHub account can access the installation.
ctxpipe does not keep this authorization, and it does not link your GitHub
account to your ctxpipe user.
### Choose the repositories
Give the App access to the repository for the knowledge and to the code
repositories that you want to link. You can start with a few repositories,
and add more later in GitHub.
### Create a Workspace and link code
[Create a Workspace](/docs/workspaces/create-workspace), then
[link code repositories](/docs/workspaces/linked-repositories) in its
**Settings**.
## What GitHub enables
* Workspace repositories that ctxpipe can read and write.
* Code search and extraction across the linked repositories.
* The docs, ADRs, `AGENTS.md`, and skills in linked repositories, as context
for agents.
* Pull requests for source changes, upgrades, and Workspace chat.
* A mirror of the issues and the closed pull requests of each linked
repository.
## Closed pull requests
When you link a GitHub repository, ctxpipe also mirrors its closed pull
requests, merged or not, into `github/pulls///` in the Workspace
repository. Each file holds the conversation, the reviews, and the changed
paths, but not the diff. It records `merged` and `mergedAt`.
* **When you link** the repository, ctxpipe backfills up to 200 of its most
recently updated closed pull requests.
* **After that**, each pull request that closes, and each later review or
comment, arrives when GitHub reports it.
* **Each Workspace** that links the repository gets its own copy.
* **When you unlink**, new pull requests stop. The mirrored files stay until
you delete them.
Draft and open pull requests are not mirrored. There is no setting for this.
## Issues
ctxpipe also mirrors the issues of each linked repository, open and closed. Each
issue is one file at `github/issues///.md`, with its
description, labels, assignees, state, and its 100 most recent comments.
* **When you link** the repository, ctxpipe backfills up to 200 of its most
recently updated issues.
* **After that**, each change to an issue, and each comment on it, arrives
when GitHub reports it. When someone deletes an issue or moves it to another
repository, ctxpipe removes its file.
The GitHub App needs the **Issues: Read-only** permission. Until the owner of
your GitHub installation accepts that permission, ctxpipe does not mirror
issues. Pull requests and code continue to work. When the owner accepts it,
ctxpipe backfills the issues of each linked repository of that account.
## In the graph
A pull request links to its repository, to the GitHub issues whose URL it
contains, and to the Linear issues that it names. A merged pull request also links to
each file that it added, changed, removed, or renamed. A pull request
**resolves** an issue when its branch names the issue. It also resolves an
issue when a closing word, such as `fixes` or `closes`, comes before the issue
identifier. A GitHub issue links to the pull requests that closed it.
# Linear connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/linear)
The **Linear connector** mirrors selected Linear teams and projects into a
[Workspace](/docs/workspaces) repository. ctxpipe requests read-only access. It
does not create or change Linear issues.
On a self-hosted deployment, the operator registers a Linear OAuth app first.
See [Self-host Linear](/docs/self-hosting/linear).
## What is mirrored
A team syncs its issues, projects, cycles, and labels. A project syncs its
issues from all teams. The mirror can include:
* issues and their comments, with the parent issue, the project milestone, and
the relations to other issues (blocks, duplicate, related);
* project and initiative updates;
* cycles, labels, and the users that the content mentions;
* customer requests, unless you clear **Include customer requests**;
* attachments and embedded media, copied into git.
ctxpipe does not copy GitHub pull requests, commits, or diffs through Linear.
It keeps only the link. The GitHub connector already indexes that content.
Customer requests can contain the names and feedback of your customers. Each
person who can read the Workspace repository can read them. Review the scope
before you merge the pull request.
## Connect Linear
Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace).
For Linear:
1. On **Connectors**, select **Add connection**, then select **Connect** on the
**Linear** row. Authorize the ctxpipe app in the Linear popup.
2. In **Add to Workspaces**, select the Workspaces, then the teams and
projects. Clear **Include customer requests** to leave them out.
3. Merge the pull request that adds `sources/linear/.md`.
To change the Workspaces or the scope later, choose **Manage** on the connector
card.
## Source file
```markdown
---
provider: linear
connection: con_example
teams: [ENG, OPS]
customerRequests: exclude
---
Engineering and operations work. Customer requests stay in the support Workspace.
```
## Files in the repository
Mirrored files are under `linear/`. Each team has a folder
`linear/teams//`, and each project has a folder
`linear/projects//`. Below them are folders such as `issues/`, `documents/`,
`cycles/`, `labels/`, `users/`, and `customer-requests/`. The paths use stable
Linear ids, so a renamed issue or project does not make a second copy.
Attachments are in an `assets/` folder next to the Markdown file. Each asset
can be up to 25 MiB, and each Linear item up to 100 MiB in total. A file
that is too large, not safe, or not available stays a link to Linear.
## Check the setup
1. The Linear workspace shows on **Connectors**.
2. After the merge, the Workspace repository has files under `linear/`.
3. When you change a mirrored issue in Linear, the file of that issue changes. This
step tests the Linear webhook.
For problems, see [Troubleshooting](/docs/connections#troubleshooting).
# Notion connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/notion)
The **Notion connector** mirrors selected pages and databases into a
[Workspace](/docs/workspaces) repository. ctxpipe reads Notion content. It does
not create or change it.
On a self-hosted deployment, the operator registers a Notion OAuth app and its
webhook first. See [Self-host Notion](/docs/self-hosting/notion).
## What is mirrored
A page syncs with all its child pages, and with the child pages that you add later. A
database syncs with all its rows. The mirror includes:
* pages as Markdown, with headings, lists, tasks, quotes, and code. Links to
other mirrored pages become relative links;
* each database as an index page, a `table.csv` snapshot, and one Markdown file
for each row;
* images, files, video, audio, and PDFs, copied into git.
Page comments are not mirrored.
ctxpipe can list only the pages and databases that are shared with the
ctxpipe integration. Share them in the authorization popup, or later in
Notion. Each person who can read the Workspace repository can read the
mirrored files.
## Connect Notion
Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace).
For Notion:
1. On **Connectors**, select **Add connection**, then select **Connect** on the
**Notion** row. Authorize your Notion workspace, and share the pages and databases that ctxpipe can
read.
2. In **Add to Workspaces**, select the Workspaces, then search for the pages
and databases. Keep the selection narrow when the Notion workspace has
private notes.
3. Merge the pull request that adds `sources/notion/.md`.
## Source file
```markdown
---
provider: notion
connection: con_example
pages:
- id: 1f2e3d4c-5b6a-4978-8695-a4b3c2d1e0f9
title: Engineering handbook
databases:
- id: 0a1b2c3d-4e5f-4061-8728-394a5b6c7d8e
title: Runbooks
---
```
## Files in the repository
* Pages: `notion/pages/--/index.md`, with child pages inside the
folder of their parent.
* Databases: `notion/databases/--/index.md`, `table.csv`, and
`rows/--/index.md`.
The paths use stable Notion ids, so a renamed page does not make a second copy.
Assets are in an `assets/` folder next to their page. Each asset can be up to
25 MiB, and each page or row up to 100 MiB in total. A file that is too large or
not safe stays a link to Notion. ctxpipe never writes the download URLs
of Notion to git, because they expire.
When a page changes in Notion, ctxpipe syncs the whole page tree or database
that holds it again.
## Check the setup
1. The Notion workspace shows on **Connectors**.
2. After the merge, the Workspace repository has files under `notion/`.
3. When you change a mirrored page in Notion, the file of that page changes. This
step tests the Notion webhook.
## Troubleshooting
* **No pages or databases show.** Share them with the ctxpipe integration in
Notion, then search again. ctxpipe lists only the pages and databases that
you share with the integration. A page that you stop sharing leaves the
list.
For other problems, see [Troubleshooting](/docs/connections#troubleshooting).
# PagerDuty connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/pagerduty)
The **PagerDuty connector** mirrors the incidents of selected services into a
[Workspace](/docs/workspaces) repository. ctxpipe requests read access and the
permission to create its webhook subscription. It does not acknowledge or
resolve incidents, and it does not show live on-call state.
After you authorize an account, ctxpipe creates one signed webhook subscription
for the connection. On a self-hosted deployment, the operator registers a
PagerDuty OAuth app first. See [Self-host PagerDuty](/docs/self-hosting/pagerduty).
## What is mirrored
ctxpipe syncs the incidents of each selected service from the last 90 days. Each
incident is one Markdown file with:
* the title, status, urgency, service, assignees, and times;
* the body of the alert that started it, with its details, such as a host
name;
* a short list of the other alerts on the incident;
* the incident notes;
* image attachments, copied next to the file.
ctxpipe removes routing keys and other secret alert fields before it writes to
git. If an incident moves to a service that is not selected, ctxpipe deletes its
file.
## Connect PagerDuty
Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace).
For PagerDuty:
1. On **Connectors**, select **Add connection**, then select **Connect** on the
**PagerDuty** row. Authorize the app in the PagerDuty popup.
2. In **Add to Workspaces**, select the Workspaces, then the services. If the
list is empty, create a service in PagerDuty first.
3. Merge the pull request that adds `sources/pagerduty/.md`.
## Source file
```markdown
---
provider: pagerduty
connection: con_example
services: [PSVC001, PSVC002]
---
```
## Files in the repository
```text
pagerduty/services//incidents/--.md
pagerduty/services//incidents/--/assets/…
```
When you remove a service from the scope, the next sync deletes the incident
files of that service.
## How incidents stay current
PagerDuty sends a signed event when an incident changes, or when a new alert
arrives on it. The subscription covers the whole account, but ctxpipe fetches
and writes only the incidents of the services in a **Connected** source. It ignores
the other events.
## Check the setup
1. The PagerDuty account shows on **Connectors**.
2. After the merge, the Workspace repository has files under
`pagerduty/services/`.
3. When you trigger or update an incident of a selected service, its file
changes. This step tests the webhook.
## Troubleshooting
* **Changes do not arrive.** Make sure that the incident is on a selected
service. Then ask the operator to check that the connection still has its
webhook subscription, and that `POST /api/v1/webhook/pagerduty` is reachable.
* **You reconnect PagerDuty.** ctxpipe keeps or replaces the webhook
subscription. It never adds a second one.
* **You remove the connector.** ctxpipe also deletes the webhook subscription
in PagerDuty.
For other problems, see [Troubleshooting](/docs/connections#troubleshooting).
# Plain connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/plain)
The **Plain connector** writes Help Center articles and support threads from
[Plain](https://www.plain.com) into a [Workspace's](/docs/workspaces) GitHub
repository. ctxpipe then indexes them like other knowledge files. Agents can read
your support documentation and see what customers report. A source file in the
Workspace holds the scope. A pull request adds it, so a person reviews the
scope before content syncs.
ctxpipe only reads from Plain. It does not change threads, articles, or
customers. The one write is a webhook target for thread activity, which ctxpipe
adds and removes.
## What is mirrored
You select Help Centers, thread labels, or both, for each Workspace.
* **Help Center:** one file for each published article, with its group and
its body as Markdown. Images are text stubs in this version.
* **Threads:** one file for each of the 100 newest threads with a label. A
thread file has the ref, status, priority, labels, the customer name, and
the first 50 messages in order, each as a quote. Internal notes are marked.
A longer thread says that Plain has more messages.
ctxpipe writes people by name only. In thread text, ctxpipe removes email
addresses, phone numbers in common formats, and links that carry a credential.
ctxpipe
keeps only `https:` links. A link with text keeps the text and loses the
link. A bare URL that is not `https:` becomes `[link removed]`. Other
personal data in the text can remain.
Thread messages are text from your customers, and they can hold personal
data. Git keeps every version of a file, so a removed thread stays in the
history of the repository. Select labels that hold the threads your
Workspace needs, and keep the repository private.
## Prerequisites
* A Plain admin who can add a machine user and an API key.
* A [Workspace](/docs/workspaces) whose repository the ctxpipe GitHub App can
write to. The repository must use format 2 or higher. If it does not, the
dialog shows **Needs the format upgrade first.** next to the Workspace. See
[Upgrades](/docs/workspaces/settings#upgrades).
* Permission to merge the pull request that adds the source.
## Where the scope lives
The connection holds the API key only. The scope is in a source file in each
Workspace, at `sources/plain/.md`. The front matter holds the Help
Center ids and the label type ids. The body is an optional description of what
the source is for.
```markdown
---
provider: plain
connection: con_example
helpCenters: [hc_01EXAMPLE]
labels: [lt_01EXAMPLE]
---
Product documentation and bug reports for the support Workspace.
```
The scope goes live after the merge. See
[Workspace settings](/docs/workspaces/settings#sources) for draft scope and
live scope.
A Workspace can have many Plain source files, and each Workspace has its own
scope.
## Setup flow
### Create a machine user and an API key
In Plain, open **Settings → Machine Users** and add a machine user for
ctxpipe. Add an API key with only these permissions:
`helpCenter:read`, `helpCenterArticle:read`,
`helpCenterArticleGroup:read`, `labelType:read`, `label:read`,
`thread:read`, `timeline:read`, `customer:read`, `user:read`,
`machineUser:read`, `webhookTarget:create`, and `webhookTarget:delete`.
Copy the key. Plain shows it one time.
### Connect Plain
In ctxpipe, open **Connectors**, select **Add connection**, then select
**Connect** on the **Plain** row. Paste the key and select **Connect Plain**.
ctxpipe checks the key and its permissions, keeps it encrypted, and adds one webhook target for
thread activity.
### Add to Workspaces
The next step is **Add to Workspaces**. Select the Workspaces that sync
from Plain. Then select the Help Centers and labels. When you select more
than one Workspace, choose **Same scope for each Workspace** or **A
different scope for each Workspace**.
Choose **Add to Workspaces**. You can add the connection to a Workspace
later. Expand the connector on **Connectors**. When no Workspace uses the
connection, choose **Add to a Workspace**. Otherwise, choose **Manage**.
### Merge the pull requests
ctxpipe opens one pull request for each Workspace. Each pull request adds
`sources/plain/.md`. The dialog shows **Merge the pull request** (or
**Merge the pull requests**) with a link to each one. Review the selected
Help Centers and labels, and the repository audience, then merge.
After each merge, the dialog shows **Pull request merged**. The source does
not sync before the merge.
### Wait for the first sync
After GitHub reports the merge, ctxpipe reads the source file and writes the
first Plain files. The Workspace indexes the new files after that. See
[How indexing works](/docs/workspaces/indexing) for the lifecycle.
## Change what syncs
Edit, rename, or remove a source in the **Settings** of the Workspace. See
[Workspace settings](/docs/workspaces/settings#sources). When you remove a
Help Center or a label from the scope, ctxpipe deletes its files in a sync
commit.
To change the Workspaces of the connection, use **Manage** on the connector
card. **Manage** opens the same **Add to Workspaces** step. It starts from the
current Workspaces and scopes.
## Files in the repository
```text
plain/help-centers//--.md
plain/threads/labels//.md
```
The source owns these folders. Do not edit the files. ctxpipe writes them again
on each sync. When an article is unpublished, or a thread leaves the newest
100 of its label, ctxpipe deletes its file in the same commit.
## Keeping content current
* **Thread webhook:** a new message, a note, or a label change on a thread
starts a sync of the labels at the end of a 10-minute window. More events in
the same window start no more syncs.
* **Daily sync:** ctxpipe syncs each connection once a day. Help Center articles
use this sync, because Plain sends no article events.
* **Sync now:** an owner or admin can select **Sync now** in the menu of the
connector card.
## Verify the complete flow
1. Connect Plain and confirm that the name of the Plain workspace appears on
**Connectors**.
2. Choose **Add to Workspaces**, select a Workspace, and select at least one
Help Center or label.
3. Merge the pull request that adds `sources/plain/.md`.
4. Confirm that the Workspace repository has files under `plain/`.
5. Add a note to a real thread with a selected label, not a Plain test
thread. Confirm that the thread file changes after the 10-minute window.
6. Confirm that the Workspace index is ready in its **Settings**.
Steps 3 and 4 test the GitHub push webhook and the first sync. Step 5 tests
the Plain webhook delivery.
## Troubleshooting
### A test thread does not sync
Plain shows a test thread only to the person who created it, so the API key
cannot read it. To test the thread sync, send an email to your support
address, and add a selected label to the new thread.
### A label or a Help Center does not sync
Plain refuses a read when the key does not have a permission in the list
above. The source can still show OK. Add that permission to the key in
Plain, then select **Sync now** in the menu of the connector card.
### The setup names missing permissions
The key does not have every permission in the list. Create a new key on the
machine user with all of them, and paste it.
### Add to a Workspace shows an error
The dialog shows a message about the format, the write access, or your role.
See [Troubleshooting](/docs/connections#troubleshooting).
### No pull request appears
Confirm that the GitHub App has write access to the Workspace repository. If
the dialog reports an error, close it and try again.
### The source shows a status other than OK
Read the status and the reason of the row in **Settings**. See
[Source status](/docs/workspaces/settings#source-status).
When a selected Help Center or label is deleted or archived in Plain, ctxpipe
tries it in three sync cycles and keeps its files. Then the source shows
**Out of access**. Remove the Help Center or the label from the source.
### The card says that threads sync once a day
Plain did not accept the webhook target. Plain sends webhooks only to a
public HTTPS address, and a workspace can have at most 25 targets. Select
**Reconnect Plain** on the card and paste the key again to try again, or use
the daily sync and **Sync now**. A reconnect replaces the target, so it also
helps when someone deleted the target in Plain.
### The card shows “Needs reconnect”
Plain refused the saved key, for example because someone deleted it.
ctxpipe keeps the files and stops the sync. Select **Reconnect Plain** on
the card and paste a new key. A permission that the key does not have does
not show here. See
[A label or a Help Center does not sync](#a-label-or-a-help-center-does-not-sync).
### Removing the connector
Choose **Remove connector** in the menu of the connector card. See
[How content stays current](/docs/connections#how-content-stays-current) for
what stays in Git. ctxpipe also deletes the webhook target in Plain while the key works. When the
key no longer works, delete the target in **Settings → Webhooks** in Plain.
### More than one ctxpipe target in Plain
A failed request can leave an extra ctxpipe webhook target in Plain. ctxpipe
refuses its events. Delete each ctxpipe target in **Settings → Webhooks** in
Plain except the newest, or delete all of them and select **Reconnect Plain**.
# Slack connector (https://docs.ctxpipe.ai/docs/connections/source-connectors/slack)
The **Slack connector** keeps the Slack threads that matter, such as decisions,
incidents, and design debates, in a [Workspace](/docs/workspaces). It does not
mirror channels or read Slack in the background. A person mentions the bot in
a thread, and ctxpipe saves that thread as Markdown.
The pull request that adds the channels to a Workspace is the reviewed part.
Each capture after that is a commit, so git history records who captured which
thread, and when.
A Slack workspace can connect to one ctxpipe organization. On a self-hosted
deployment, the operator creates the Slack app first. See
[Self-host Slack](/docs/self-hosting/slack).
## Slack connector or Slack MCP
| Tool | Use it to |
| ------------------------------------------------------------------ | ------------------------------------------------------- |
| **ctxpipe Slack connector** (this page) | Keep chosen threads in git, where ctxpipe indexes them. |
| **[Slack MCP server](https://docs.slack.dev/ai/slack-mcp-server)** | Search and read live Slack from your coding agent. |
The two do different jobs. You can use both.
## Connect Slack
Follow [Add a tool to a Workspace](/docs/connections#add-a-tool-to-a-workspace).
For Slack:
1. On **Connectors**, select **Add connection**, then select **Connect** on the
**Slack** row. Authorize the Slack workspace.
2. In **Add to Workspaces**, select the Workspaces, then the channels for each
one. When captures can include private channels, use a separate Workspace
with a private repository.
3. Merge the pull request that adds `sources/slack/.md`. Captures start
after the merge.
## Capture a thread
1. Invite the bot to the channel with `/invite`. The connector card shows the
**@handle** of the bot.
2. In the thread, mention the bot:
```text
@your-bot-handle capture this
```
A mention with no other text also captures. A mention outside a thread does
not capture.
ctxpipe replies in the thread with **ctxpipe agent working…**, then changes the
reply to **Engineering context captured into**, with a link to the file in each
Workspace that has the channel. ctxpipe does not answer other questions in
Slack. It replies with what it can do.
Later replies in the thread are not captured by themselves. Mention the bot
again to capture the thread again. The new capture replaces the file.
## What is captured
ctxpipe saves the thread in each Workspace whose **Connected** Slack source has the
channel, with one commit for each Workspace:
* the file is
`slack/channels//threads////thread.md`;
* the front matter holds the capture time, the Slack link, the channel and
thread ids, and the handle and name of the person who captured it;
* the authors and the people mentioned show by name;
* files and embedded media are copied into the `assets/` folder of the thread.
Each asset can be up to 25 MiB, and each thread up to 100 MiB in total;
* a thread longer than 500 messages keeps its oldest 500, and the reply in Slack
says so.
Not included: channel mirrors, history backfill, direct messages, and automatic
updates after a capture. When you uninstall Slack, or remove a channel from a
source, the captured threads stay in the repository.
## Troubleshooting
* **A mention does nothing.** Check that the connector shows **Connected**,
that a **Connected** source has the channel, and that the bot is in the channel.
Use Slack autocomplete to mention the bot. If no Workspace has the channel,
the bot replies "Nothing captured: no Workspace has this channel."
* **A private channel does not work.** Invite the bot to the channel first.
* **The source is OK, but the bot is not in the channel.** The reason on the
row tells you to invite the bot.
For other problems, see [Troubleshooting](/docs/connections#troubleshooting).