Optimize OpenRouter performance and latency. Use when reducing response times or improving throughput. Trigger with phrases like 'openrouter performance', 'openrouter latency', 'speed up openrouter', 'openrouter optimization'.
npxskills add jeremylongshore/claude-code-plugins-plus-skills--skill openrouter-performance-tuningLoading…
OpenRouter adds minimal overhead (~50-100ms) to direct provider calls. Most latency comes from the upstream model. Key levers: model selection (smaller = faster), streaming (lower TTFT), parallel requests, prompt size reduction, and provider routing to faster infrastructure. This skill covers benchmarking, streaming optimization, concurrent processing, and connection tuning.
sk-or-v1-...) exported as OPENROUTER_API_KEY — see the openrouter-install-auth skill for setupopenai package) — the examples use both the sync OpenAI client and AsyncOpenAI for parallel processinganthropic/claude-3.5-sonnet; a :free model is enough to validate the benchmark harness itselfHTTP-Referer / X-Title header values for your app (set in every client constructor here)benchmark_model() from Benchmark Latency against your candidate models (e.g. openai/gpt-4o-mini vs anthropic/claude-3.5-sonnet) and record p50/p95.stream_completion() per Streaming for Lower TTFT and verify ttft_ms drops (typically 2-10x).parallel_completions() per Parallel Request Processing, capping concurrency with asyncio.Semaphore (max_concurrent=5-10).timeout=30.0 and max_retries=2 instead of a new client per request.max_tokens, shrink prompts, consider :nitro variants and provider routing), then re-run the benchmark to quantify each change.import os, time, statistics
from openai import OpenAI
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)
def benchmark_model(model: str, prompt: str = "Say hello", n: int = 5) -> dict:
"""Benchmark a model's latency over N requests."""
latencies = []
for _ in range(n):
start = time.monotonic()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
max_tokens=50,
)
latencies.append((time.monotonic() - start) * 1000)
return {
"model": model,
"p50_ms": round(statistics.median(latencies)),
"p95_ms": round(sorted(latencies)[int(len(latencies) * 0.95)]),
"avg_ms": round(statistics.mean(latencies)),
"min_ms": round(min(latencies)),
"max_ms": round(max(latencies)),
}
# Compare fast vs slow models
for model in ["openai/gpt-4o-mini", "anthropic/claude-3-haiku", "anthropic/claude-3.5-sonnet"]:
result = benchmark_model(model)
print(f"{result['model']}: p50={result['p50_ms']}ms p95={result['p95_ms']}ms")
def stream_completion(messages, model="openai/gpt-4o-mini", **kwargs):
"""Stream response for lower time-to-first-token."""
start = time.monotonic()
first_token_time = None
full_content = []
stream = client.chat.completions.create(
model=model, messages=messages, stream=True,
stream_options={"include_usage": True}, # Get token counts at end
**kwargs,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
if first_token_time is None:
first_token_time = (time.monotonic() - start) * 1000
full_content.append(chunk.choices[0].delta.content)
total_time = (time.monotonic() - start) * 1000
return {
"content": "".join(full_content),
"ttft_ms": round(first_token_time or 0),
"total_ms": round(total_time),
}
import asyncio
from openai import AsyncOpenAI
async def parallel_completions(prompts: list[str], model="openai/gpt-4o-mini",
max_concurrent=10, **kwargs):
"""Process multiple prompts concurrently."""
semaphore = asyncio.Semaphore(max_concurrent)
client = AsyncOpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)
async def process(prompt):
async with semaphore:
response = await client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
**kwargs,
)
return response.choices[0].message.content
return await asyncio.gather(*[process(p) for p in prompts])
# 10 requests in parallel instead of sequential
results = asyncio.run(parallel_completions(
["Summarize: " + text for text in documents],
max_concurrent=5,
max_tokens=200,
))
| Optimization | Impact | Effort |
|---|---|---|
| Use streaming | TTFT drops 2-10x | Low |
| Use smaller models for simple tasks | 2-5x faster | Low |
| Reduce prompt size | Proportional to reduction | Medium |
Set max_tokens | Caps response time | Low |
| Parallel requests | N requests in ~1 request time | Medium |
Use :nitro variant | Faster inference (where available) | Low |
| Provider routing to fastest | 10-30% latency reduction | Low |
| Connection keep-alive | Saves TCP/TLS handshake | Low |
| Speed | Models | Typical TTFT |
|---|---|---|
| Fastest | openai/gpt-4o-mini, anthropic/claude-3-haiku | 200-500ms |
| Fast | openai/gpt-4o, google/gemini-2.0-flash-001 | 500ms-1s |
| Standard | anthropic/claude-3.5-sonnet | 1-3s |
| Slow | openai/o1, reasoning models | 5-30s |
# Reuse client instance (connection pooling)
# BAD: creating new client per request
for prompt in prompts:
c = OpenAI(base_url="https://openrouter.ai/api/v1", ...) # New TCP connection each time
c.chat.completions.create(...)
# GOOD: reuse single client
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
timeout=30.0, # Set appropriate timeout
max_retries=2, # Built-in retry with backoff
default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)
for prompt in prompts:
client.chat.completions.create(...) # Reuses HTTP connection
benchmark_model(): p50_ms, p95_ms, avg_ms, min_ms, max_ms over N sample requestsstream_completion(): the full content plus ttft_ms and total_ms for each requestparallel_completions() produced in roughly one request's wall-clock time instead of N sequential round-tripsBenchmark two fastest-tier candidates before committing to one:
for model in ["openai/gpt-4o-mini", "anthropic/claude-3-haiku"]:
r = benchmark_model(model, n=5)
print(f"{r['model']}: p50={r['p50_ms']}ms p95={r['p95_ms']}ms avg={r['avg_ms']}ms")
# openai/gpt-4o-mini: p50=430ms p95=610ms avg=455ms
# anthropic/claude-3-haiku: p50=395ms p95=580ms avg=418ms
Both land in the fastest tier (200-500ms typical TTFT), so choose on cost or quality — then stream_completion() cuts perceived latency further for user-facing paths. More worked examples: references/examples.md.
| Error | Cause | Fix |
|---|---|---|
| High TTFT (>5s) | Model cold-starting or overloaded | Switch to :nitro variant or different provider |
| Timeout errors | max_tokens too high or model too slow | Reduce max_tokens; use streaming; increase timeout |
| Throughput bottleneck | Sequential processing | Use async + semaphore for concurrent requests |
| Inconsistent latency | Provider load varies | Use provider.order to pin to fastest provider |
max_tokens on every request to bound response time and costasyncio.Semaphore to control concurrency and avoid overwhelming the API:nitro model variants for latency-critical paths
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).