Configuration

Model configuration

LLM tiers, embedding models, and OpenAI-compatible providers when self-hosting ctxpipe.

When you self-host ctxpipe, you set the LLM and embedding models with environment variables. The service supports any OpenAI-compatible HTTP API. Use MODEL_PROVIDER to select OpenRouter-specific features, Azure (api-key auth), or Bedrock (native SDK with IAM SigV4).

MODEL_PROVIDER values

ValueAuth (chat + embeddings)MODEL_PROVIDER_URL defaultNotes
openai-likeAuthorization: Bearerhttps://openrouter.ai/api/v1Generic compatible API; no OpenRouter-only extras.
openrouterAuthorization: BearersameEnables plugins, prompt cache, reasoning flags, medium-tier model fallbacks.
azureapi-key header onlyrequiredSet your Azure OpenAI OpenAI-compatible base URL; override model name env vars for deployment IDs.
bedrockIAM SigV4 (ECS task role)not requiredNative Bedrock Runtime via AWS SDK. Set MODEL_BEDROCK_AWS_REGION or rely on AWS_REGION / AWS_DEFAULT_REGION. No MODEL_PROVIDER_API_KEY on AWS CDK deployments.

If you do not set MODEL_PROVIDER, it defaults to openai-like. To keep the previous behavior (OpenRouter URL heuristics that added extras whenever the host was OpenRouter), set MODEL_PROVIDER=openrouter.

For Vertex and other gateways that expect Bearer and a base URL, use openai-like. If you route through OpenRouter, use openrouter.

AWS CDK Bedrock deployments

When you deploy with @ctxpipe/aws-cdk and modelProvider.kind: "bedrock", the construct does not store a static MODEL_PROVIDER_API_KEY. Backend and worker tasks receive IAM permissions (bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream) and call Bedrock Runtime through the AWS SDK. They use the ECS task role (SigV4). Enable your chosen model IDs in the Bedrock console for the stack region before you use them. If the account is in an AWS Organization, also confirm that service control policies (SCPs) allow those same foundation-model ARNs. An SCP explicit deny overrides the task role. It shows as AccessDeniedException … explicit deny in a service control policy.

Bedrock embeddings (Cohere only)

When MODEL_PROVIDER=bedrock, embeddings use the native Bedrock Runtime API. Only Cohere embed models work. Other Bedrock embedding models do not work yet. On AWS CDK, the default is cohere.embed-v4:0 when you do not set models.embedding. You can also use an inference-profile ID, for example global.cohere.embed-v4:0. If an SCP denies a profile ID, the error often names the base …/foundation-model/cohere.embed-v4:0 ARN.

Quick Start (OpenRouter)

The default setup uses OpenRouter. Set your API key and start the stack:

MODEL_PROVIDER_API_KEY=sk-or-v1-... docker compose --profile deploy up -d

LLM and embeddings both use OpenRouter by default. You do not need extra configuration.

Environment Variables

VariableRequiredDefaultDescription
MODEL_PROVIDERNoopenai-likeopenai-like | openrouter | azure | bedrock - see table above.
MODEL_PROVIDER_API_KEYYes*-API key for OpenAI-compatible providers. Not used for native Bedrock (MODEL_PROVIDER=bedrock).
MODEL_PROVIDER_URLNo**OpenRouter base when not azureBase URL for OpenAI-compatible LLM/embeddings HTTP. Required for azure. Not required for native Bedrock.
MODEL_BEDROCK_AWS_REGIONNoAWS_REGION / stack regionAWS region for Bedrock Runtime when MODEL_PROVIDER=bedrock.
MODEL_FAST_NAMENoopenai/gpt-6-luna?reasoning.effort=highLLM model for the fast tier (OpenRouter slash ids).
MODEL_MEDIUM_NAMENoopenai/gpt-6-luna?reasoning.effort=xhighLLM model for the medium tier.
MODEL_HIGH_NAMENoxiaomi/mimo-v2.6-proLLM model for the high tier.
MODEL_EMBEDDING_NAMENoopenai/text-embedding-3-largeEmbedding model ID.

*Required for each provider except bedrock. The backend refuses model calls without it. For a provider that needs no key, such as Ollama, set any value that is not empty.

**Default MODEL_PROVIDER_URL is https://openrouter.ai/api/v1 for openai-like and openrouter so bundled default model IDs stay valid.

How to Pick Models

LLM tiers

The service uses three tiers for different workloads:

TierUse caseDefault model spec
fastQuick tasks, naming, planningopenai/gpt-6-luna?reasoning.effort=high
mediumMain agent, balanced cost/qualityopenai/gpt-6-luna?reasoning.effort=xhigh
highComplex reasoning, best qualityxiaomi/mimo-v2.6-pro

Override any tier with MODEL_FAST_NAME, MODEL_MEDIUM_NAME, or MODEL_HIGH_NAME. Use model IDs from your provider (OpenRouter: org/model-name; Bedrock on AWS CDK: vendor.model with dot, e.g. openai.gpt-6-luna?reasoning.effort=high).

Model spec query params

Append ?reasoning.effort=value to set the reasoning depth of a tier without extra environment variables. The supported values are none, minimal, low, medium, high, and xhigh. Each provider adapter maps this value to its upstream API (reasoning_effort on OpenAI-like / Azure / Bedrock Chat Completions; nested reasoning.effort on OpenRouter).

openai/gpt-6-luna?reasoning.effort=high
openai.gpt-6-luna?reasoning.effort=xhigh

To disable reasoning in environment specs, use reasoning.effort=none. Do not use reasoning=false. Call sites can also pass getModel(..., { reasoning: false }), which merges reasoning.effort=none over the tier spec. ctxpipe sends model ids upstream without query strings.

When MODEL_PROVIDER=openrouter (or hosted ctxpipe, which sets it), the medium tier sends the OpenRouter models array. If the primary model fails (rate limits, downtime, moderation), routing tries the next distinct model IDs in order (MODEL_FAST_NAME, then MODEL_HIGH_NAME). ctxpipe removes duplicate base model ids from that chain. This does not apply to openai-like.

Embedding model

Important: The embedding model must support 2000 dimensions. The schema stores vectors in vector(2000); a model that is not compatible fails.

  • OpenAI-compatible providers (OpenRouter, OpenAI, Vertex): Use models that accept a dimensions parameter, for example openai/text-embedding-3-large (3072 dims, truncated to 2000).
  • Amazon Bedrock: Use a Cohere embed model ID (for example cohere.embed-v4:0 or global.cohere.embed-v4:0). Other Bedrock embed families are not supported yet.
  • Ollama: Use models with native 2000-dim support, e.g. qwen3-embedding. Set MODEL_PROVIDER_URL=http://host:11434/v1 and MODEL_EMBEDDING_NAME=qwen3-embedding (embeddings use {MODEL_PROVIDER_URL}/embeddings).

All providers use the same OpenAI-compatible embeddings API. Ollama exposes it at /v1/embeddings.

Provider Examples

OpenRouter (default host)

MODEL_PROVIDER_API_KEY=sk-or-v1-...
# Optional explicit mode (enables OpenRouter extras; default URL is OpenRouter either way):
# MODEL_PROVIDER=openrouter
MODEL_PROVIDER_URL=https://openrouter.ai/api/v1

Azure OpenAI

MODEL_PROVIDER=azure
MODEL_PROVIDER_API_KEY=<your-azure-api-key>
MODEL_PROVIDER_URL=https://YOUR_RESOURCE.openai.azure.com/openai/deployments/YOUR_DEPLOYMENT
# Set model env vars to your deployment names as needed.

Amazon Bedrock (native SDK)

MODEL_PROVIDER=bedrock
MODEL_BEDROCK_AWS_REGION=us-east-1
MODEL_EMBEDDING_NAME=cohere.embed-v4:0  # or global.cohere.embed-v4:0; Cohere only on Bedrock
# ECS task role / instance profile supplies AWS credentials; no MODEL_PROVIDER_URL or API key.
# Org SCPs must allow the underlying foundation-model ARN (see AWS CDK Bedrock callout above).

Ollama (local chat + embeddings)

MODEL_PROVIDER_URL=http://localhost:11434/v1
MODEL_PROVIDER_API_KEY=ollama        # Ollama needs no key, but the value must not be empty
MODEL_FAST_NAME=<your-chat-model>
MODEL_MEDIUM_NAME=<your-chat-model>
MODEL_HIGH_NAME=<your-chat-model>
MODEL_EMBEDDING_NAME=qwen3-embedding
# For Docker: use http://host.docker.internal:11434/v1

Set each model name. The defaults are OpenRouter model ids, which Ollama does not know.

Resources