Providers

This guide explains how to configure provider-backed model access for package examples, FastAPI apps, and source demos. It does not compare model quality or recommend one provider over another.

This guide is for developers who want to run real hosted model calls from SwarmForge. It assumes that you know which provider you want to use and can set environment variables or pass configuration in code.

After reading this guide, you should be able to:

• set the required environment variables for your integration path
• configure ModelConfig(...)
• understand which variables are used by package examples, FastAPI apps, and the demo UI

The provider layer exposes one OpenAI-compatible client wrapper and one configuration object. Every runnable runtime example in these docs uses this layer, so you should explicitly choose both the provider and the model before you run anything.

Environment Variables By Service

Use the lists below instead of relying on .env.example being open locally.

Package runtime examples

Applies to Getting Started, Create Your First Agent, Create Your First Multi-Agent Swarm, and Orchestration.

Always set:

• MODEL_PROVIDER
• LLM_MODEL

Then set the provider-specific auth variable for the provider you chose:

• OpenRouter: OPENROUTER_API_KEY
• Gemini: GEMINI_API_KEY or GOOGLE_API_KEY
• OpenAI-compatible openai: OPENAI_API_KEY
• Any other provider name such as anthropic-proxy: ${PROVIDER_NAME_UPPER}_API_KEY

Optional OpenRouter attribution headers:

• OPENROUTER_SITE_URL
• OPENROUTER_APP_NAME
• OPENROUTER_FALLBACK_MODELS comma-separated fallback model slugs

Optional OpenRouter latency/cost tuning:

• OPENROUTER_SERVICE_TIER "priority" or "flex" (OpenAI models only)

FastAPI server defaults

Applies to API, examples/fastapi_swarm.py, and examples/fastapi_server.py.

FastAPI reads provider configuration from the server environment or from default_model_config in your app wiring. Set the same provider variables as the package runtime examples:

• MODEL_PROVIDER
• LLM_MODEL
• matching API key variable for that provider

Optional server bind variables:

• SWARMFORGE_HOST default: 127.0.0.1
• SWARMFORGE_PORT default: 8000

Demo UI

Applies to the local demo-ui/ app when it is connected to examples/fastapi_swarm.py.

The demo UI reads only these variables from the root .env:

• MODEL_PROVIDER optional, defaults the provider selector
• LLM_MODEL optional, defaults the model field
• SWARMFORGE_HOST optional, defaults API host to 127.0.0.1
• SWARMFORGE_PORT optional, defaults API port to 8000

The demo UI does not read provider API keys directly. Those stay on the server side.

OpenRouter conversation script

Applies to examples/openrouter_conversation.py.

Required:

• MODEL_PROVIDER=openrouter
• LLM_MODEL
• OPENROUTER_API_KEY
• OPENROUTER_SITE_URL
• OPENROUTER_APP_NAME

Supported Modes

• OpenRouter through https://openrouter.ai/api/v1
• Gemini OpenAI-compatible mode through https://generativelanguage.googleapis.com/v1beta/openai/
• other OpenAI-compatible endpoints by overriding base_url, api_key, and model

OpenRouter

Copy the repository example config first:

cp .env.example .env

Then set the OpenRouter values in .env:

MODEL_PROVIDER=openrouter
LLM_MODEL=openrouter/auto
OPENROUTER_API_KEY=sk-or-...
OPENROUTER_SITE_URL=https://your-app.example
OPENROUTER_APP_NAME="Your App Name"
OPENROUTER_FALLBACK_MODELS=openai/gpt-4o-mini,anthropic/claude-3.5-sonnet

Shell exports still work, but .env is the default path used by the examples and tests. Replace LLM_MODEL with the exact OpenRouter model slug you want when you need stable behavior instead of router-selected defaults.

Default OpenRouter settings:

• provider: openrouter
• model: openrouter/auto
• base URL: https://openrouter.ai/api/v1

Gemini OpenAI-compat

Set Gemini values in .env:

MODEL_PROVIDER=gemini
LLM_MODEL=gemini-3-flash-preview
GEMINI_API_KEY=...

GOOGLE_API_KEY is also supported if you prefer that variable name.

Gemini uses:

• provider: gemini
• base URL: https://generativelanguage.googleapis.com/v1beta/openai/

OpenAI-compatible Custom Provider

If you are not using OpenRouter or Gemini, set:

MODEL_PROVIDER=openai
LLM_MODEL=gpt-4.1-mini
OPENAI_API_KEY=...

If you construct ModelConfig(...) directly in code, you can also override base_url there for another OpenAI-compatible endpoint.

Configuration Surface

ModelConfig supports:

• provider
• base_url
• api_key
• model
• temperature
• max_tokens
• site_url
• app_name
• default_headers
• default_chat_params
• fallback_models
• cache_enabled (OpenRouter only)
• cache_ttl (OpenRouter only)
• prompt_caching (OpenRouter only)
• prompt_cache_ttl (OpenRouter only)
• service_tier (OpenRouter/OpenAI only)

Minimal client setup

from swarmforge.env import require_env_vars
from swarmforge.evaluation.provider import ModelConfig, OpenAIClientWrapper

env = require_env_vars("MODEL_PROVIDER", "LLM_MODEL")
client = OpenAIClientWrapper(ModelConfig())
response = client.chat_completion(
	messages=[
		{"role": "system", "content": "You are a concise assistant."},
		{
			"role": "user",
			"content": (
				"Reply with one sentence that names the active provider and model: "
				f"{env['MODEL_PROVIDER']} / {env['LLM_MODEL']}"
			),
		},
	]
)
print(response.choices[0].message.content)

Switch providers by changing .env values rather than editing the code sample.

For tool and handoff examples, choose a model that supports tool or function calling.

Response Caching (OpenRouter)

OpenRouter offers a response caching feature that caches identical API request responses server-side. Cache hits return immediately with zero billing and lower latency — the request never reaches the provider.

How it works

Two requests are considered identical when they share the same API key, model, endpoint type, streaming mode, and request body (including all parameters). OpenRouter generates a cache key from these inputs. Cache is scoped to your API key — different keys don't share cache.

Default TTL is 300 seconds (5 minutes), configurable from 1 to 86400 seconds (24 hours).

Enabling via environment

OPENROUTER_CACHE_ENABLED=true
OPENROUTER_CACHE_TTL=600

Enabling in code

from swarmforge.evaluation.provider import ModelConfig

config = ModelConfig(
    provider="openrouter",
    model="openrouter/auto",
    cache_enabled=True,
    cache_ttl=600,  # optional, defaults to 300
)

Passing cache_enabled=False disables caching even if the env var is set. Omitting it (or passing None) falls back to the env var.

The X-OpenRouter-Cache and X-OpenRouter-Cache-TTL headers are injected automatically into every request made by the OpenAIClientWrapper when caching is enabled.

Cache clearing

To force a fresh response for a specific request, pass X-OpenRouter-Cache-Clear: true via extra_headers or default_headers:

config = ModelConfig(
    cache_enabled=True,
    default_headers={"X-OpenRouter-Cache-Clear": "true"},
)

Limitations

• Cache is scoped to your API key — rotating the key empties the cache
• Concurrent identical requests arriving before the first response is cached both result in a miss
• Available only for OpenRouter provider
• Not available with account-level Zero Data Retention (ZDR)

Prompt Caching (OpenRouter)

Prompt caching is a provider-level feature that reduces cost when repeated prompt prefixes (e.g. system instructions, long context) are shared across requests. Unlike response caching, this is not free — cached tokens are billed at a discount — but OpenRouter uses provider sticky routing to maximize hit rates.

Provider behavior

Provider	Requires config	Notes
OpenAI	No	Automatic, minimum 1024 tokens
DeepSeek	No	Automatic
Gemini 2.5	Yes (prompt_caching)	Needs cache_control breakpoints in messages
Anthropic Claude	Yes (prompt_caching)	Needs cache_control at request level or per-block
Grok	No	Automatic

For Anthropic models, enabling prompt_caching injects a top-level cache_control into the request body, which tells OpenRouter to apply automatic caching — the system advances cache breakpoints forward as conversation history grows. This is ideal for multi-turn conversations.

For OpenAI and DeepSeek, prompt caching is automatic regardless of this flag (when prompts are long enough).

Enabling via environment

OPENROUTER_PROMPT_CACHING=true
OPENROUTER_PROMPT_CACHE_TTL=1h    # optional: "1h" for 1-hour TTL (default 5min)

Enabling in code

from swarmforge.evaluation.provider import ModelConfig

config = ModelConfig(
    provider="openrouter",
    prompt_caching=True,
    prompt_cache_ttl="1h",  # optional, default is 5-minute TTL
)

The cache_control field is merged into extra_body of every request made by the OpenAIClientWrapper when prompt caching is enabled.

Explicit cache breakpoints

For fine-grained control (e.g. caching only specific content blocks), pass cache_control directly in message content via default_chat_params. This is useful when you want to cache a large system prompt but keep user messages dynamic:

config = ModelConfig(
    prompt_caching=False,  # disable automatic injection
    default_chat_params={
        "extra_body": {
            "cache_control": {"type": "ephemeral"}
        }
    },
)

For Gemini explicit breakpoints, insert cache_control inside message content blocks — though this requires constructing the messages array manually rather than relying on automatic injection.

Notes

• Prompt caching and response caching can be used together; they operate at different layers
• Provider sticky routing ensures subsequent requests hit the same provider endpoint for warm caches
• Cache hit metrics appear in usage.prompt_tokens_details.cached_tokens in API responses

Latency Optimization

Time-to-first-token (TTFT) depends on provider routing, model selection, request size, and client configuration. These are the highest-ROI levers available in the SDK.

Pin your model

openrouter/auto evaluates available providers on every request to pick the cheapest or fastest route. This adds 100–300ms of routing overhead. Pin to a specific model slug to skip evaluation:

# Instead of this (adds routing latency):
LLM_MODEL=openrouter/auto

# Use this (direct provider+model):
LLM_MODEL=openai/gpt-4o
LLM_MODEL=anthropic/claude-sonnet-4

Service tier (OpenAI models only)

OpenRouter supports a service_tier parameter that controls queue priority. "priority" reduces provider-side queue wait at higher cost; "flex" reduces cost at potentially higher latency.

OPENROUTER_SERVICE_TIER=priority

Or in code:

config = ModelConfig(
    provider="openrouter",
    service_tier="priority",
)

Accepted values: auto, default, flex, priority. Only applies to OpenAI provider models.

Enable response caching

Cache identical requests server-side. Hits return with zero billing and no provider round-trip:

OPENROUTER_CACHE_ENABLED=true
OPENROUTER_CACHE_TTL=600

Enable prompt caching

Cache repeated prompt prefixes (system instructions, long context) at the provider level. OpenAI and DeepSeek do this automatically; Anthropic and Gemini require explicit opt-in:

OPENROUTER_PROMPT_CACHING=true

Client timeout

The SDK now sets a default 30-second timeout on all HTTP calls (configurable at construction). Requests that hang will fail fast instead of blocking for the SDK default of 10 minutes:

from swarmforge.evaluation.provider import ModelConfig, OpenAIClientWrapper

client = OpenAIClientWrapper(
    ModelConfig(),
    timeout=15.0,   # seconds
    max_retries=1,
)

Connection reuse

The OpenAIClientWrapper configures HTTP connection pooling with keep-alive so subsequent requests reuse the same TCP connection instead of opening a new one for every call.

Wrapper Behavior

OpenAIClientWrapper.chat_completion(...) passes through:

• chat messages
• optional tool definitions
• optional timeout override
• optional structured response_format
• optional provider-specific extra_params

Notes

• .env values are loaded automatically from the current working directory upward
• provider-specific headers are applied for OpenRouter attribution
• OpenRouter can retry with fallback models from fallback_models or OPENROUTER_FALLBACK_MODELS
• extra OpenAI-compatible request fields can be passed through default_chat_params
• explicit api_key values support ${ENV_VAR} substitution