Native API Key Strategy — Investigation Report

This document records the findings from the native API key strategy investigation (June 2026). It answers whether SwarmForge should keep its unified OpenAI-compatible client for native OpenAI and Gemini keys, or adopt per-provider SDKs—with emphasis on multi-turn tool calling.

For setup instructions, see Providers.

Executive Summary

Provider	Recommendation	Rationale
OpenAI (native key)	Keep unified OpenAI SDK	Already the native path (api.openai.com/v1). No separate SDK needed.
Gemini (native key)	Keep OpenAI-compat for simple tool loops; plan native adapter if Gemini 3+ agents or Google built-in tools are required	OpenAI-compat works for basic scenarios; Google recommends google-genai for agentic multi-turn flows.
OpenRouter	Keep as primary multi-provider path	Normalizes provider quirks; all probe scenarios passed for both OpenAI and Gemini model slugs.

Architecture decision: Option 1 — Keep unified OpenAI-compatible layer with targeted provider quirks centralized in the turn runner and conversation runner. Add a native google-genai adapter (Option 2) only when Gemini 3 thought signatures, built-in Google tools, or Interactions API state are first-class requirements.

Current Architecture

SwarmForge routes all hosted model calls through one stack:

ModelConfig → OpenAIClientWrapper (openai SDK) → OpenAICompatibleTurnRunner → Orchestrator

Provider	Env key	Base URL
OpenAI	OPENAI_API_KEY	https://api.openai.com/v1
Gemini	GEMINI_API_KEY / GOOGLE_API_KEY	https://generativelanguage.googleapis.com/v1beta/openai/
OpenRouter	OPENROUTER_API_KEY	https://openrouter.ai/api/v1

Tools are always defined in OpenAI tools format (src/swarmforge/swarm/tools.py). There is no google-genai, Anthropic, or LiteLLM dependency.

Probe Methodology

The probe script scripts/provider_probe.py exercises seven scenarios per provider:

1. Single-turn chat
2. Single tool call + follow-up (role: tool)
3. Multi-turn tool loop via ConversationRunner
4. Parallel tool calls in one assistant turn
5. Streaming with tools
6. Structured output (Pydantic .parse())
7. Orchestrator tool loop via process_swarm_stream

Run locally:

python3 scripts/provider_probe.py
python3 scripts/provider_probe.py --provider gemini --model gemini-3.1-flash-lite
python3 scripts/provider_probe.py --output /tmp/probe-results.json

Compatibility Matrix Results

Probe run: 2026-06-29 (results in scripts/provider_probe_results.json).

Scenario	OpenAI direct	Gemini direct	OpenRouter openai/gpt-4o-mini	OpenRouter google/gemini-3.1-flash-lite
single_turn_chat	skipped (no key)	skipped (no key)	pass	pass
single_tool_call_role_tool	skipped	skipped	pass	pass
multi_turn_tool_loop_role_tool	skipped	skipped	pass (after fix)	pass
parallel_tool_calls	skipped	skipped	pass	pass
streaming_with_tools	skipped	skipped	pass	pass
structured_output	skipped	skipped	pass	pass
orchestrator_tool_loop	skipped	skipped	pass	pass

Note: Native OPENAI_API_KEY and GEMINI_API_KEY were not available in the probe environment. OpenRouter results proxy OpenAI and Gemini behavior. Re-run with native keys before production sign-off on direct Gemini endpoints.

Pre-fix failure (resolved)

ConversationRunner multi-turn tool loops failed on strict OpenAI providers with:

Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.

Root cause: Tool result messages were appended to conversation_history without first persisting the assistant message that issued tool_calls. The orchestrator already did this correctly; only the evaluation ConversationRunner was affected.

Fix: Append assistant tool_calls to history before tool results (src/swarmforge/evaluation/runner/conversation.py).

Issue Classification

Issue	Severity	Class	Status
Missing assistant tool_calls in ConversationRunner history	Blocker (OpenAI strict)	A — Fixable in unified layer	Fixed
Runtime-only tool fields sent to Gemini (mock_response, handler)	Blocker (Gemini strict)	A — Fixable in unified layer	Fixed in OpenAIClientWrapper via sanitize_tool_definitions
Inconsistent Gemini tool-result role (function vs tool)	Medium	A — Fixable in unified layer	Open: ConversationRunner uses role: function for Gemini; orchestrator and turn runner use role: tool
Empty assistant content when tool_calls present	Medium (Gemini direct)	A — Fixable	Open: orchestrator uses content: response_text or None; may need "[TOOL CALL]" placeholder for native Gemini
Gemini 3 thought_signature round-trip	High (Gemini 3+ agents)	A partial / B native SDK	Fixed for ConversationRunner and orchestrator via serialize_assistant_message_for_history
tools_format=gemini authoring vs runtime	Low	A — Doc only	Confirmed: authoring emits FunctionDeclaration shape; runtime expects OpenAI function.name
Built-in Google tools (Search, MCP, code execution)	N/A today	B — Requires native SDK	Out of scope for unified layer
Streaming tool-call deltas	Low	A — Works	Passed for OpenRouter OpenAI and Gemini slugs

Classification key:

• A — Fixable in unified OpenAI-compat layer (message shaping, passthrough fields)
• B — Requires google-genai provider adapter or Interactions API
• C — Fundamental OpenAI-compat limitation; use OpenRouter proxy or native SDK

Google OpenAI-Compatibility vs Native SDK

Per Gemini OpenAI compatibility:

Factor	Unified OpenAI-compat	Native google-genai
Google's recommendation for new work	Bridge for existing OpenAI code	Preferred for agents and multi-turn tools
Maturity	Beta; unlisted params silently ignored	GA SDK
Basic function calling	Supported	Supported
Multi-turn tool loops	Works with message-order fixes	SDK handles response objects automatically
Gemini 3 thought signatures	Manual extra_content.google.thought_signature	Automatic when appending full model response
Built-in tools	Limited / extra_body	First-class

For OpenAI native keys, there is no compat-vs-native tradeoff—the OpenAI SDK against api.openai.com is already native.

Architecture Options — Scored

Criterion (weight)	Option 1: Unified OpenAI-compat	Option 2: Provider adapter	Option 3: OpenRouter for non-OpenAI
Multi-turn tool reliability (high)	8/10 after ConversationRunner fix	9/10	9/10
Single tool schema everywhere (high)	10/10	6/10	8/10
Gemini feature parity (medium)	5/10	9/10	7/10
Maintenance cost (medium)	7/10	5/10	8/10
Native key requirement (low)	10/10	10/10	3/10
Weighted score	8.0	7.2	7.5

Option 1 — Keep unified OpenAI-compat (recommended)

Fix provider quirks centrally in turn_runners.py, conversation.py, and orchestrator message persistence. One SDK, one tool format.

When sufficient: OpenAI direct keys, Gemini simple tool loops, OpenRouter multi-provider routing.

Option 2 — Provider adapter layer

Keep OpenAI-compat for OpenAI/OpenRouter; add google-genai behind the existing AgentTurnRunner protocol.

When needed: Gemini 3+ multi-turn agents, thought signatures, built-in Google tools, Interactions API state.

Option 3 — Hybrid routing

Direct OpenAI keys; route Gemini/Anthropic through OpenRouter.

When needed: Production swarms where direct Gemini key quirks are unacceptable but native keys for OpenAI are required.

Recommended Patch List

Priority	Patch	File(s)
Done	Strip runtime-only tool fields before API requests	evaluation/provider/client.py
Done	Persist assistant tool_calls before tool results in evaluation runner	evaluation/runner/conversation.py
High	Centralize Gemini tool-result role (function vs tool) in turn runner	swarm/turn_runners.py
Medium	Non-empty assistant content when persisting tool_calls for Gemini	swarm/orchestrator.py
Medium	Capture and replay thought_signature from assistant responses	evaluation/provider/messages.py, conversation.py, turn_runners.py, orchestrator.py
Low	Document that tools_format=gemini is authoring-output only	docs/authoring.md, docs/providers.md

Test Coverage

Env-gated integration tests live in tests/test_provider_compatibility_integration.py. They skip automatically when provider API keys are missing.

Unit test for ConversationRunner tool-history ordering: tests/test_conversation_runner.py.

References

• Gemini OpenAI compatibility
• Gemini function calling
• Thought signatures
• Google GenAI SDK libraries
• SwarmForge provider setup: providers.md