Execute distribute requests across multiple OpenRouter configurations. Use when scaling or implementing geographic distribution. Trigger with phrases like 'openrouter load balance', 'distribute requests', 'openrouter scaling', 'multi-key openrouter'.
npxskills add jeremylongshore/claude-code-plugins-plus-skills--skill openrouter-load-balancingLoading…
A single OpenRouter API key has rate limits (requests/minute and tokens/minute). To scale beyond those limits, distribute requests across multiple keys. OpenRouter also provides server-side load balancing via provider routing and the :nitro variant for low-latency inference. This skill covers multi-key rotation, health-based routing, circuit breakers, and concurrent request patterns.
OPENROUTER_KEY_1, OPENROUTER_KEY_2, OPENROUTER_KEY_3 so the KeyPool has keys to rotate — see the openrouter-install-auth skill for creating and exporting keysOPENROUTER_API_KEY exported for the single-key concurrent-processing patternrequests () — the concurrent example uses from the same packagepip install openai requestsAsyncOpenAIGET /api/v1/auth/keyKeyPool from Multi-Key Round Robin — it round-robins across keys, trips a circuit breaker after 3 consecutive errors, and auto-recovers a key after a 60s cooldown.balanced_completion(): on RateLimitError it calls pool.mark_error(key) and retries with the next healthy key.parallel_completions() from Concurrent Request Processing — an asyncio.Semaphore (max_concurrent=3-5) caps in-flight requests against a single key.extra_body={"provider": {"order": [...], "allow_fallbacks": True}} so OpenRouter spreads the same model across Anthropic, AWS Bedrock, and GCP Vertex.check_rate_limits() (GET /api/v1/auth/key) from Rate Limit Awareness, and when 429s hit all keys simultaneously, apply the fixes in Error Handling (more keys, request queuing).import os, itertools, time, logging
from openai import OpenAI, RateLimitError
from dataclasses import dataclass, field
log = logging.getLogger("openrouter.lb")
@dataclass
class KeyPool:
"""Round-robin API key pool with health tracking."""
keys: list[str]
_cycle: itertools.cycle = field(init=False, repr=False)
_health: dict[str, dict] = field(init=False, default_factory=dict)
def __post_init__(self):
self._cycle = itertools.cycle(self.keys)
self._health = {k: {"errors": 0, "last_error": 0, "healthy": True} for k in self.keys}
def next_key(self) -> str:
"""Get next healthy key."""
attempts = 0
while attempts < len(self.keys):
key = next(self._cycle)
h = self._health[key]
# Recover after 60s cooldown
if not h["healthy"] and time.time() - h["last_error"] > 60:
h["healthy"] = True
h["errors"] = 0
if h["healthy"]:
return key
attempts += 1
# All keys unhealthy -- return any and hope for the best
return next(self._cycle)
def mark_error(self, key: str):
h = self._health[key]
h["errors"] += 1
h["last_error"] = time.time()
if h["errors"] >= 3: # Circuit breaker: 3 errors → unhealthy
h["healthy"] = False
log.warning(f"Key {key[:12]}... marked unhealthy after {h['errors']} errors")
def mark_success(self, key: str):
self._health[key]["errors"] = 0
self._health[key]["healthy"] = True
pool = KeyPool(keys=[
os.environ.get("OPENROUTER_KEY_1", ""),
os.environ.get("OPENROUTER_KEY_2", ""),
os.environ.get("OPENROUTER_KEY_3", ""),
])
def balanced_completion(messages, model="anthropic/claude-3.5-sonnet", **kwargs):
"""Send request using next healthy key from the pool."""
key = pool.next_key()
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=key,
default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)
try:
response = client.chat.completions.create(
model=model, messages=messages, **kwargs
)
pool.mark_success(key)
return response
except RateLimitError:
pool.mark_error(key)
# Retry with next key
return balanced_completion(messages, model, **kwargs)
import asyncio
from openai import AsyncOpenAI
async def parallel_completions(prompts: list[str], model="openai/gpt-4o-mini",
max_concurrent=5, **kwargs):
"""Process multiple prompts concurrently with rate limiting."""
semaphore = asyncio.Semaphore(max_concurrent)
client = AsyncOpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)
async def process_one(prompt: str):
async with semaphore:
response = await client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
**kwargs,
)
return response.choices[0].message.content
return await asyncio.gather(*[process_one(p) for p in prompts])
# Usage
results = asyncio.run(parallel_completions(
["Summarize X", "Translate Y", "Analyze Z"],
max_concurrent=3,
max_tokens=500,
))
# OpenRouter can distribute across providers for the same model
response = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=200,
extra_body={
"provider": {
# Let OpenRouter pick the best available provider
"order": ["Anthropic", "AWS Bedrock", "GCP Vertex"],
"allow_fallbacks": True,
},
},
)
import requests
def check_rate_limits(api_key: str) -> dict:
"""Check current rate limit status for a key."""
resp = requests.get(
"https://openrouter.ai/api/v1/auth/key",
headers={"Authorization": f"Bearer {api_key}"},
)
data = resp.json()["data"]
return {
"requests_limit": data["rate_limit"]["requests"],
"interval": data["rate_limit"]["interval"],
"credits_used": data["usage"],
"credits_limit": data.get("limit"),
}
# Check all keys in pool
for key in pool.keys:
limits = check_rate_limits(key)
print(f"Key {key[:12]}...: {limits}")
healthy flags, and log lines like Key sk-or-v1-abc... marked unhealthy after 3 errorsparallel_completions() — one per input prompt, gathered concurrentlycheck_rate_limits(): requests_limit, interval, credits_used, credits_limitSix requests through a two-key pool split evenly, and the pool's stats confirm the distribution:
for i in range(6):
balanced_completion(f"Request {i}: Hello!")
print(pool.get_stats())
# {'sk-or-v1-abc': {'requests': 3, 'errors': 0},
# 'sk-or-v1-def': {'requests': 3, 'errors': 0}}
Zero errors means no key tripped the circuit breaker; a nonzero errors count on one key with requests skewing to the other shows health-based routing doing its job. More worked examples: references/examples.md.
| Error | Cause | Fix |
|---|---|---|
| 429 on all keys | All keys rate-limited simultaneously | Add more keys; implement request queuing |
| Uneven load distribution | Round-robin not accounting for in-flight requests | Use weighted distribution based on current load |
| Key health false positive | Transient error marked key unhealthy | Use sliding window (3 errors in 60s) before marking unhealthy |
| Concurrent request failures | Too many parallel requests | Reduce semaphore limit; add backoff |
asyncio.Semaphore to control concurrency and prevent overwhelming the API
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).