Model configuration
LLM tiers, embedding models, and OpenAI-compatible providers when self-hosting ctx|.
When self-hosting ctx|, you configure LLM and embedding models via environment variables. The service supports any OpenAI-compatible HTTP API. Use MODEL_PROVIDER to select OpenRouter-specific features, Azure (api-key auth), or Bedrock (native SDK with IAM SigV4).
MODEL_PROVIDER values
| Value | Auth (chat + embeddings) | MODEL_PROVIDER_URL default | Notes |
|---|---|---|---|
openai-like | Authorization: Bearer | https://openrouter.ai/api/v1 | Generic compatible API; no OpenRouter-only extras. |
openrouter | Authorization: Bearer | same | Enables plugins, prompt cache, reasoning flags, medium-tier model fallbacks. |
azure | api-key header only | required | Set your Azure OpenAI OpenAI-compatible base URL; override model name env vars for deployment IDs. |
bedrock | IAM SigV4 (ECS task role) | not required | Native Bedrock Runtime via AWS SDK. Set MODEL_BEDROCK_AWS_REGION or rely on AWS_REGION / AWS_DEFAULT_REGION. No MODEL_PROVIDER_API_KEY on AWS CDK deployments. |
If MODEL_PROVIDER is unset, it defaults to openai-like. To keep previous behavior (OpenRouter URL heuristics that added extras whenever the host was OpenRouter), set MODEL_PROVIDER=openrouter.
Vertex and other gateways that expect Bearer + base URL use openai-like (or openrouter if routed via OpenRouter).
AWS CDK Bedrock deployments
When you deploy with @ctxpipe/aws-cdk and modelProvider.kind: "bedrock", the construct does not store a static MODEL_PROVIDER_API_KEY. Backend and worker tasks receive IAM permissions (bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream) and call Bedrock Runtime through the AWS SDK using the ECS task role (SigV4). Enable your chosen model IDs in the Bedrock console for the stack region before use.
Bedrock embeddings (Cohere only)
When MODEL_PROVIDER=bedrock, embeddings use the native Bedrock Runtime API and currently support Cohere embed models only (default cohere.embed-v4:0 on AWS CDK when models.embedding is omitted). Non-Cohere Bedrock embedding models are not supported yet.
Quick Start (OpenRouter)
The default setup uses OpenRouter. Set your API key and start the stack:
MODEL_PROVIDER_API_KEY=sk-or-v1-... docker compose --profile deploy up -dLLM and embeddings both use OpenRouter by default. No extra configuration needed.
Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
MODEL_PROVIDER | No | openai-like | openai-like | openrouter | azure | bedrock - see table above. |
MODEL_PROVIDER_API_KEY | Usually* | - | API key for OpenAI-compatible providers. Not used for native Bedrock (MODEL_PROVIDER=bedrock) on AWS CDK. |
MODEL_PROVIDER_URL | No** | OpenRouter base when not azure | Base URL for OpenAI-compatible LLM/embeddings HTTP. Required for azure. Not required for native Bedrock. |
MODEL_BEDROCK_AWS_REGION | No | AWS_REGION / stack region | AWS region for Bedrock Runtime when MODEL_PROVIDER=bedrock. |
MODEL_FAST_NAME | No | openai/gpt-5.5?reasoning.effort=low | LLM model for the fast tier (OpenRouter slash ids). |
MODEL_MEDIUM_NAME | No | openai/gpt-5.5?reasoning.effort=medium | LLM model for the medium tier. |
MODEL_HIGH_NAME | No | openai/gpt-5.5?reasoning.effort=high | LLM model for the high tier. |
MODEL_EMBEDDING_NAME | No | openai/text-embedding-3-large | Embedding model ID. |
*Required for LLM when not using Bedrock IAM-only.
**Default MODEL_PROVIDER_URL is https://openrouter.ai/api/v1 for openai-like and openrouter so bundled default model IDs stay valid.
How to Pick Models
LLM tiers
The service uses three tiers for different workloads:
| Tier | Use case | Default model spec |
|---|---|---|
| fast | Quick tasks, naming, planning | openai/gpt-5.5?reasoning.effort=low |
| medium | Main agent, balanced cost/quality | openai/gpt-5.5?reasoning.effort=medium |
| high | Complex reasoning, best quality | openai/gpt-5.5?reasoning.effort=high |
Override any tier with MODEL_FAST_NAME, MODEL_MEDIUM_NAME, or MODEL_HIGH_NAME. Use model IDs from your provider (OpenRouter: org/model-name; Bedrock on AWS CDK: vendor.model with dot, e.g. openai.gpt-5.5?reasoning.effort=low).
Model spec query params
Append ?reasoning.effort=value to encode per-tier reasoning depth without extra env vars. Supported values: none, minimal, low, medium, high, xhigh. Each provider adapter maps this to its upstream API (reasoning_effort on OpenAI-like / Azure / Bedrock Chat Completions; nested reasoning.effort on OpenRouter).
openai/gpt-5.5?reasoning.effort=low
openai.gpt-5.5?reasoning.effort=highUse reasoning.effort=none (not reasoning=false) to disable reasoning in env specs. Call sites can also pass getModel(..., { reasoning: false }), which merges reasoning.effort=none over the tier spec. Model ids are sent upstream without query strings.
When MODEL_PROVIDER=openrouter (or fully managed deployments that set it), the medium tier sends OpenRouter’s models array so that if the primary model errors (rate limits, downtime, moderation), routing tries the next distinct model IDs in order (MODEL_FAST_NAME, then MODEL_HIGH_NAME). Duplicate base model ids in that chain are removed automatically. This does not apply to openai-like.
Embedding model
Important: The embedding model must support 2000 dimensions. The schema stores vectors in vector(2000); incompatible models will fail.
- OpenAI-compatible providers (OpenRouter, OpenAI, Vertex): Use models that accept a
dimensionsparameter, e.g.openai/text-embedding-3-large(3072 dims, truncates to 2000). - Amazon Bedrock: Use a Cohere embed model ID (for example
cohere.embed-v4:0). Other Bedrock embed families are not supported yet. - Ollama: Use models with native 2000-dim support, e.g.
qwen3-embedding. SetMODEL_PROVIDER_URL=http://host:11434/v1andMODEL_EMBEDDING_NAME=qwen3-embedding(embeddings use{MODEL_PROVIDER_URL}/embeddings).
All providers use the same OpenAI-compatible embeddings API; Ollama exposes it at /v1/embeddings.
Provider Examples
OpenRouter (default host)
MODEL_PROVIDER_API_KEY=sk-or-v1-...
# Optional explicit mode (enables OpenRouter extras; default URL is OpenRouter either way):
# MODEL_PROVIDER=openrouter
MODEL_PROVIDER_URL=https://openrouter.ai/api/v1Azure OpenAI
MODEL_PROVIDER=azure
MODEL_PROVIDER_API_KEY=<your-azure-api-key>
MODEL_PROVIDER_URL=https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT
# Set model env vars to your deployment names as needed.Amazon Bedrock (native SDK)
MODEL_PROVIDER=bedrock
MODEL_BEDROCK_AWS_REGION=us-east-1
MODEL_EMBEDDING_NAME=cohere.embed-v4:0 # Cohere embed models only on Bedrock
# ECS task role / instance profile supplies AWS credentials; no MODEL_PROVIDER_URL or API key.Ollama (local chat + embeddings)
MODEL_PROVIDER_URL=http://localhost:11434/v1
MODEL_EMBEDDING_NAME=qwen3-embedding
# For Docker: use http://host.docker.internal:11434/v1