# Model configuration (https://docs.ctxpipe.ai/docs/self-hosting/models)

When you self-host ctxpipe, you set the LLM and embedding models with environment variables. The service supports any OpenAI-compatible HTTP API. Use **`MODEL_PROVIDER`** to select OpenRouter-specific features, Azure (`api-key` auth), or Bedrock (native SDK with IAM SigV4).

## `MODEL_PROVIDER` values

| Value         | Auth (chat + embeddings)  | `MODEL_PROVIDER_URL` default   | Notes                                                                                                                                                                  |
| ------------- | ------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `openai-like` | `Authorization: Bearer`   | `https://openrouter.ai/api/v1` | Generic compatible API; no OpenRouter-only extras.                                                                                                                     |
| `openrouter`  | `Authorization: Bearer`   | same                           | Enables plugins, prompt cache, reasoning flags, medium-tier model fallbacks.                                                                                           |
| `azure`       | `api-key` header only     | **required**                   | Set your Azure OpenAI OpenAI-compatible base URL; override model name env vars for deployment IDs.                                                                     |
| `bedrock`     | IAM SigV4 (ECS task role) | not required                   | Native Bedrock Runtime via AWS SDK. Set `MODEL_BEDROCK_AWS_REGION` or rely on `AWS_REGION` / `AWS_DEFAULT_REGION`. No `MODEL_PROVIDER_API_KEY` on AWS CDK deployments. |

If you do not set **`MODEL_PROVIDER`**, it defaults to **`openai-like`**. To keep the **previous** behavior (OpenRouter URL heuristics that added extras whenever the host was OpenRouter), set **`MODEL_PROVIDER=openrouter`**.

For Vertex and other gateways that expect Bearer and a base URL, use **`openai-like`**. If you route through OpenRouter, use **`openrouter`**.

<Callout type="info" title="AWS CDK Bedrock deployments">
  When you deploy with [`@ctxpipe/aws-cdk`](/docs/self-hosting/deployment/aws) and `modelProvider.kind: "bedrock"`, the construct does **not** store a static `MODEL_PROVIDER_API_KEY`. Backend and worker tasks receive IAM permissions (`bedrock:InvokeModel`, `bedrock:InvokeModelWithResponseStream`) and call Bedrock Runtime through the AWS SDK. They use the ECS task role (SigV4). Enable your chosen model IDs in the Bedrock console for the stack region before you use them. If the account is in an AWS Organization, also confirm that **service control policies (SCPs)** allow those same foundation-model ARNs. An SCP explicit deny overrides the task role. It shows as `AccessDeniedException` … `explicit deny in a service control policy`.
</Callout>

<Callout type="warn" title="Bedrock embeddings (Cohere only)">
  When `MODEL_PROVIDER=bedrock`, embeddings use the native Bedrock Runtime API. Only **Cohere** embed models work. Other Bedrock embedding models do not work yet. On AWS CDK, the default is `cohere.embed-v4:0` when you do not set `models.embedding`. You can also use an inference-profile ID, for example `global.cohere.embed-v4:0`. If an SCP denies a profile ID, the error often names the base `…/foundation-model/cohere.embed-v4:0` ARN.
</Callout>

## Quick Start (OpenRouter)

The default setup uses OpenRouter. Set your API key and start the stack:

```bash
MODEL_PROVIDER_API_KEY=sk-or-v1-... docker compose --profile deploy up -d
```

LLM and embeddings both use OpenRouter by default. You do not need extra configuration.

## Environment Variables

| Variable                   | Required | Default                                    | Description                                                                                                                                                                                                                                                                                                                                                              |
| -------------------------- | -------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `MODEL_PROVIDER`           | No       | `openai-like`                              | `openai-like` \| `openrouter` \| `azure` \| `bedrock` - see table above.                                                                                                                                                                                                                                                                                                 |
| `MODEL_PROVIDER_API_KEY`   | Yes\*    | -                                          | API key for OpenAI-compatible providers. Not used for native Bedrock (`MODEL_PROVIDER=bedrock`).                                                                                                                                                                                                                                                                         |
| `MODEL_PROVIDER_URL`       | No\*\*   | OpenRouter base when not `azure`           | Base URL for OpenAI-compatible LLM/embeddings HTTP. **Required** for `azure`. Not required for native Bedrock.                                                                                                                                                                                                                                                           |
| `MODEL_BEDROCK_AWS_REGION` | No       | `AWS_REGION` / stack region                | AWS region for Bedrock Runtime when `MODEL_PROVIDER=bedrock`.                                                                                                                                                                                                                                                                                                            |
| `MODEL_PROVIDER_ALLOWLIST` | No       | -                                          | OpenRouter provider slugs, separated by commas (for example `openai,anthropic,azure`). When set, each OpenRouter call goes only to these providers. When unset, OpenRouter can use any provider. It needs `MODEL_PROVIDER=openrouter` or a `MODEL_PROVIDER_URL` on `openrouter.ai`. A value that is not a slug list, or other model settings, stop the backend at start. |
| `MODEL_FAST_NAME`          | No       | `openai/gpt-6-luna?reasoning.effort=high`  | LLM model for the fast tier (OpenRouter slash ids).                                                                                                                                                                                                                                                                                                                      |
| `MODEL_MEDIUM_NAME`        | No       | `openai/gpt-6-luna?reasoning.effort=xhigh` | LLM model for the medium tier.                                                                                                                                                                                                                                                                                                                                           |
| `MODEL_HIGH_NAME`          | No       | `openai/gpt-6-luna?reasoning.effort=xhigh` | LLM model for the high tier.                                                                                                                                                                                                                                                                                                                                             |
| `MODEL_EMBEDDING_NAME`     | No       | `openai/text-embedding-3-large`            | Embedding model ID.                                                                                                                                                                                                                                                                                                                                                      |
| `MODEL_VISION_NAME`        | No       | `openai/gpt-6.1-terra`                     | The one OpenRouter model that reads the images of the knowledge plane. It must take image input and have a ZDR endpoint on OpenRouter. A value of only spaces stops the backend at start.                                                                                                                                                                                |

\*Required for each provider except `bedrock`. The backend refuses model calls without it. For a provider that needs no key, such as Ollama, set any value that is not empty.

\*\*Default `MODEL_PROVIDER_URL` is `https://openrouter.ai/api/v1` for `openai-like` and `openrouter` so bundled default model IDs stay valid.

Each model call to OpenRouter allows only zero data retention (ZDR) endpoints, with no data collection. A model with no ZDR endpoint fails. A call goes to OpenRouter when `MODEL_PROVIDER` is `openrouter`, or when `MODEL_PROVIDER_URL` is on `openrouter.ai`. Another OpenAI-compatible API does not get these routing fields.

## How to Pick Models

### LLM tiers

The service uses three tiers for different workloads:

| Tier       | Use case                          | Default model spec                         |
| ---------- | --------------------------------- | ------------------------------------------ |
| **fast**   | Quick tasks, naming, planning     | `openai/gpt-6-luna?reasoning.effort=high`  |
| **medium** | Main agent, balanced cost/quality | `openai/gpt-6-luna?reasoning.effort=xhigh` |
| **high**   | Complex reasoning, best quality   | `openai/gpt-6-luna?reasoning.effort=xhigh` |

Override any tier with `MODEL_FAST_NAME`, `MODEL_MEDIUM_NAME`, or `MODEL_HIGH_NAME`. Use model IDs from your provider (OpenRouter: `org/model-name`; Bedrock on AWS CDK: `vendor.model` with dot, e.g. `openai.gpt-6-luna?reasoning.effort=high`).

### Model spec query params

Append **`?reasoning.effort=value`** to set the reasoning depth of a tier without extra environment variables. The supported values are `none`, `minimal`, `low`, `medium`, `high`, and `xhigh`. Each provider adapter maps this value to its upstream API (`reasoning_effort` on OpenAI-like / Azure / Bedrock Chat Completions; nested `reasoning.effort` on OpenRouter).

```text
openai/gpt-6-luna?reasoning.effort=high
openai.gpt-6-luna?reasoning.effort=xhigh
```

To disable reasoning in environment specs, use **`reasoning.effort=none`**. Do not use `reasoning=false`. Call sites can also pass `getModel(..., { reasoning: false })`, which merges `reasoning.effort=none` over the tier spec. ctxpipe sends model ids upstream without query strings.

When **`MODEL_PROVIDER=openrouter`** (or hosted ctxpipe, which sets it), the **medium** tier sends the OpenRouter `models` array. If the primary model fails (rate limits, downtime, moderation), routing tries the next distinct model IDs in order (**`MODEL_FAST_NAME`**, then **`MODEL_HIGH_NAME`**). ctxpipe removes duplicate base model ids from that chain. This does **not** apply to `openai-like`.

### Embedding model

**Important:** The embedding model must support **2000 dimensions**. The schema stores vectors in `vector(2000)`; a model that is not compatible fails.

* **OpenAI-compatible providers** (OpenRouter, OpenAI, Vertex): Use models that accept a `dimensions` parameter, for example `openai/text-embedding-3-large` (3072 dims, truncated to 2000).
* **Amazon Bedrock**: Use a **Cohere** embed model ID (for example `cohere.embed-v4:0` or `global.cohere.embed-v4:0`). Other Bedrock embed families are not supported yet.
* **Ollama**: Use models with native 2000-dim support, e.g. `qwen3-embedding`. Set `MODEL_PROVIDER_URL=http://host:11434/v1` and `MODEL_EMBEDDING_NAME=qwen3-embedding` (embeddings use `{MODEL_PROVIDER_URL}/embeddings`).

All providers use the same OpenAI-compatible embeddings API. Ollama exposes it at `/v1/embeddings`.

## Provider Examples

### OpenRouter (default host)

```bash
MODEL_PROVIDER_API_KEY=sk-or-v1-...
# Optional explicit mode (enables OpenRouter extras; default URL is OpenRouter either way):
# MODEL_PROVIDER=openrouter
MODEL_PROVIDER_URL=https://openrouter.ai/api/v1
```

### Azure OpenAI

```bash
MODEL_PROVIDER=azure
MODEL_PROVIDER_API_KEY=<your-azure-api-key>
MODEL_PROVIDER_URL=https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT
# Set model env vars to your deployment names as needed.
```

### Amazon Bedrock (native SDK)

```bash
MODEL_PROVIDER=bedrock
MODEL_BEDROCK_AWS_REGION=us-east-1
MODEL_EMBEDDING_NAME=cohere.embed-v4:0  # or global.cohere.embed-v4:0; Cohere only on Bedrock
# ECS task role / instance profile supplies AWS credentials; no MODEL_PROVIDER_URL or API key.
# Org SCPs must allow the underlying foundation-model ARN (see AWS CDK Bedrock callout above).
```

### Ollama (local chat + embeddings)

```bash
MODEL_PROVIDER_URL=http://localhost:11434/v1
MODEL_PROVIDER_API_KEY=ollama        # Ollama needs no key, but the value must not be empty
MODEL_FAST_NAME=<your-chat-model>
MODEL_MEDIUM_NAME=<your-chat-model>
MODEL_HIGH_NAME=<your-chat-model>
MODEL_EMBEDDING_NAME=qwen3-embedding
# For Docker: use http://host.docker.internal:11434/v1
```

Set each model name. The defaults are OpenRouter model ids, which Ollama does
not know.

## Resources

* [OpenRouter models](https://openrouter.ai/models)
* [OpenRouter embeddings](https://openrouter.ai/docs/api/reference/embeddings)
* [Ollama OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility#%2Fv1%2Fembeddings)
