Model configuration
LLM tiers, embedding models, and OpenAI-compatible providers when self-hosting ctxpipe.
When you self-host ctxpipe, you set the LLM and embedding models with environment variables. The service supports any OpenAI-compatible HTTP API. Use MODEL_PROVIDER to select OpenRouter-specific features, Azure (api-key auth), or Bedrock (native SDK with IAM SigV4).
MODEL_PROVIDER values
| Value | Auth (chat + embeddings) | MODEL_PROVIDER_URL default | Notes |
|---|---|---|---|
openai-like | Authorization: Bearer | https://openrouter.ai/api/v1 | Generic compatible API; no OpenRouter-only extras. |
openrouter | Authorization: Bearer | same | Enables plugins, prompt cache, reasoning flags, medium-tier model fallbacks. |
azure | api-key header only | required | Set your Azure OpenAI OpenAI-compatible base URL; override model name env vars for deployment IDs. |
bedrock | IAM SigV4 (ECS task role) | not required | Native Bedrock Runtime via AWS SDK. Set MODEL_BEDROCK_AWS_REGION or rely on AWS_REGION / AWS_DEFAULT_REGION. No MODEL_PROVIDER_API_KEY on AWS CDK deployments. |
If you do not set MODEL_PROVIDER, it defaults to openai-like. To keep the previous behavior (OpenRouter URL heuristics that added extras whenever the host was OpenRouter), set MODEL_PROVIDER=openrouter.
For Vertex and other gateways that expect Bearer and a base URL, use openai-like. If you route through OpenRouter, use openrouter.
AWS CDK Bedrock deployments
When you deploy with @ctxpipe/aws-cdk and modelProvider.kind: "bedrock", the construct does not store a static MODEL_PROVIDER_API_KEY. Backend and worker tasks receive IAM permissions (bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream) and call Bedrock Runtime through the AWS SDK. They use the ECS task role (SigV4). Enable your chosen model IDs in the Bedrock console for the stack region before you use them. If the account is in an AWS Organization, also confirm that service control policies (SCPs) allow those same foundation-model ARNs. An SCP explicit deny overrides the task role. It shows as AccessDeniedException … explicit deny in a service control policy.
Bedrock embeddings (Cohere only)
When MODEL_PROVIDER=bedrock, embeddings use the native Bedrock Runtime API. Only Cohere embed models work. Other Bedrock embedding models do not work yet. On AWS CDK, the default is cohere.embed-v4:0 when you do not set models.embedding. You can also use an inference-profile ID, for example global.cohere.embed-v4:0. If an SCP denies a profile ID, the error often names the base …/foundation-model/cohere.embed-v4:0 ARN.
Quick Start (OpenRouter)
The default setup uses OpenRouter. Set your API key and start the stack:
MODEL_PROVIDER_API_KEY=sk-or-v1-... docker compose --profile deploy up -dLLM and embeddings both use OpenRouter by default. You do not need extra configuration.
Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
MODEL_PROVIDER | No | openai-like | openai-like | openrouter | azure | bedrock - see table above. |
MODEL_PROVIDER_API_KEY | Yes* | - | API key for OpenAI-compatible providers. Not used for native Bedrock (MODEL_PROVIDER=bedrock). |
MODEL_PROVIDER_URL | No** | OpenRouter base when not azure | Base URL for OpenAI-compatible LLM/embeddings HTTP. Required for azure. Not required for native Bedrock. |
MODEL_BEDROCK_AWS_REGION | No | AWS_REGION / stack region | AWS region for Bedrock Runtime when MODEL_PROVIDER=bedrock. |
MODEL_FAST_NAME | No | openai/gpt-6-luna?reasoning.effort=high | LLM model for the fast tier (OpenRouter slash ids). |
MODEL_MEDIUM_NAME | No | openai/gpt-6-luna?reasoning.effort=xhigh | LLM model for the medium tier. |
MODEL_HIGH_NAME | No | xiaomi/mimo-v2.6-pro | LLM model for the high tier. |
MODEL_EMBEDDING_NAME | No | openai/text-embedding-3-large | Embedding model ID. |
*Required for each provider except bedrock. The backend refuses model calls without it. For a provider that needs no key, such as Ollama, set any value that is not empty.
**Default MODEL_PROVIDER_URL is https://openrouter.ai/api/v1 for openai-like and openrouter so bundled default model IDs stay valid.
How to Pick Models
LLM tiers
The service uses three tiers for different workloads:
| Tier | Use case | Default model spec |
|---|---|---|
| fast | Quick tasks, naming, planning | openai/gpt-6-luna?reasoning.effort=high |
| medium | Main agent, balanced cost/quality | openai/gpt-6-luna?reasoning.effort=xhigh |
| high | Complex reasoning, best quality | xiaomi/mimo-v2.6-pro |
Override any tier with MODEL_FAST_NAME, MODEL_MEDIUM_NAME, or MODEL_HIGH_NAME. Use model IDs from your provider (OpenRouter: org/model-name; Bedrock on AWS CDK: vendor.model with dot, e.g. openai.gpt-6-luna?reasoning.effort=high).
Model spec query params
Append ?reasoning.effort=value to set the reasoning depth of a tier without extra environment variables. The supported values are none, minimal, low, medium, high, and xhigh. Each provider adapter maps this value to its upstream API (reasoning_effort on OpenAI-like / Azure / Bedrock Chat Completions; nested reasoning.effort on OpenRouter).
openai/gpt-6-luna?reasoning.effort=high
openai.gpt-6-luna?reasoning.effort=xhighTo disable reasoning in environment specs, use reasoning.effort=none. Do not use reasoning=false. Call sites can also pass getModel(..., { reasoning: false }), which merges reasoning.effort=none over the tier spec. ctxpipe sends model ids upstream without query strings.
When MODEL_PROVIDER=openrouter (or hosted ctxpipe, which sets it), the medium tier sends the OpenRouter models array. If the primary model fails (rate limits, downtime, moderation), routing tries the next distinct model IDs in order (MODEL_FAST_NAME, then MODEL_HIGH_NAME). ctxpipe removes duplicate base model ids from that chain. This does not apply to openai-like.
Embedding model
Important: The embedding model must support 2000 dimensions. The schema stores vectors in vector(2000); a model that is not compatible fails.
- OpenAI-compatible providers (OpenRouter, OpenAI, Vertex): Use models that accept a
dimensionsparameter, for exampleopenai/text-embedding-3-large(3072 dims, truncated to 2000). - Amazon Bedrock: Use a Cohere embed model ID (for example
cohere.embed-v4:0orglobal.cohere.embed-v4:0). Other Bedrock embed families are not supported yet. - Ollama: Use models with native 2000-dim support, e.g.
qwen3-embedding. SetMODEL_PROVIDER_URL=http://host:11434/v1andMODEL_EMBEDDING_NAME=qwen3-embedding(embeddings use{MODEL_PROVIDER_URL}/embeddings).
All providers use the same OpenAI-compatible embeddings API. Ollama exposes it at /v1/embeddings.
Provider Examples
OpenRouter (default host)
MODEL_PROVIDER_API_KEY=sk-or-v1-...
# Optional explicit mode (enables OpenRouter extras; default URL is OpenRouter either way):
# MODEL_PROVIDER=openrouter
MODEL_PROVIDER_URL=https://openrouter.ai/api/v1Azure OpenAI
MODEL_PROVIDER=azure
MODEL_PROVIDER_API_KEY=<your-azure-api-key>
MODEL_PROVIDER_URL=https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT
# Set model env vars to your deployment names as needed.Amazon Bedrock (native SDK)
MODEL_PROVIDER=bedrock
MODEL_BEDROCK_AWS_REGION=us-east-1
MODEL_EMBEDDING_NAME=cohere.embed-v4:0 # or global.cohere.embed-v4:0; Cohere only on Bedrock
# ECS task role / instance profile supplies AWS credentials; no MODEL_PROVIDER_URL or API key.
# Org SCPs must allow the underlying foundation-model ARN (see AWS CDK Bedrock callout above).Ollama (local chat + embeddings)
MODEL_PROVIDER_URL=http://localhost:11434/v1
MODEL_PROVIDER_API_KEY=ollama # Ollama needs no key, but the value must not be empty
MODEL_FAST_NAME=<your-chat-model>
MODEL_MEDIUM_NAME=<your-chat-model>
MODEL_HIGH_NAME=<your-chat-model>
MODEL_EMBEDDING_NAME=qwen3-embedding
# For Docker: use http://host.docker.internal:11434/v1Set each model name. The defaults are OpenRouter model ids, which Ollama does not know.