Implement response caching for OpenRouter efficiency. Use when optimizing costs or reducing latency for repeated queries. Trigger with phrases like 'openrouter cache', 'cache llm responses', 'openrouter redis', 'semantic caching'.
npxskills add jeremylongshore/claude-code-plugins-plus-skills--skill openrouter-caching-strategyLoading…
OpenRouter charges per token, so caching identical or similar requests can dramatically cut costs. Deterministic requests (temperature=0) with the same model and messages produce identical outputs -- these are safe to cache. This skill covers in-memory caching, persistent caching with TTL, and Anthropic prompt caching via OpenRouter.
sk-or-v1-...) exported as OPENROUTER_API_KEY — see the openrouter-install-auth skill for setupredis client package for the persistent cache; Node.js 18+ with the OpenAI SDK for the TypeScript variant in the referenceslocalhost:6379 for Persistent Cache with Redis (the in-memory LLMCache needs no infrastructure)temperature=0temperature=0); non-zero temperatures produce different outputs each call and must never be cached.LLMCache plus cached_completion() gives you TTL expiry and hit/miss counters in a single process.redis_cached_completion() stores results under or:<sha256> keys with r.setex TTL expiry and falls through to a direct API call on a miss.:floor), messages, temperature, max_tokens, and top_p; exclude stream and the HTTP-Referer/X-Title headers.cache_control: {"type": "ephemeral"} per Anthropic Prompt Caching via OpenRouter — cache reads bill at 0.1x the input rate.import os, hashlib, json, time
from typing import Optional
from openai import OpenAI
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=os.environ["OPENROUTER_API_KEY"],
default_headers={"HTTP-Referer": "https://my-app.com", "X-Title": "my-app"},
)
class LLMCache:
def __init__(self, ttl_seconds: int = 3600):
self._cache: dict[str, tuple[dict, float]] = {}
self._ttl = ttl_seconds
self.hits = 0
self.misses = 0
def _key(self, model: str, messages: list, **kwargs) -> str:
blob = json.dumps({"model": model, "messages": messages, **kwargs}, sort_keys=True)
return hashlib.sha256(blob.encode()).hexdigest()
def get(self, model: str, messages: list, **kwargs) -> Optional[dict]:
k = self._key(model, messages, **kwargs)
if k in self._cache:
data, ts = self._cache[k]
if time.time() - ts < self._ttl:
self.hits += 1
return data
del self._cache[k]
self.misses += 1
return None
def set(self, model: str, messages: list, response: dict, **kwargs):
k = self._key(model, messages, **kwargs)
self._cache[k] = (response, time.time())
cache = LLMCache(ttl_seconds=1800)
def cached_completion(messages, model="anthropic/claude-3.5-sonnet", **kwargs):
"""Only cache deterministic requests (temperature=0)."""
kwargs.setdefault("temperature", 0)
kwargs.setdefault("max_tokens", 1024)
cached = cache.get(model, messages, **kwargs)
if cached:
return cached
response = client.chat.completions.create(model=model, messages=messages, **kwargs)
result = {
"content": response.choices[0].message.content,
"model": response.model,
"usage": {"prompt": response.usage.prompt_tokens, "completion": response.usage.completion_tokens},
}
cache.set(model, messages, result, **kwargs)
return result
import redis, json, hashlib
r = redis.Redis(host="localhost", port=6379, db=0)
def redis_cached_completion(messages, model="openai/gpt-4o-mini", ttl=3600, **kwargs):
"""Cache in Redis with automatic TTL expiry."""
kwargs["temperature"] = 0 # Must be deterministic
key = f"or:{hashlib.sha256(json.dumps({'m': model, 'msgs': messages, **kwargs}, sort_keys=True).encode()).hexdigest()}"
cached = r.get(key)
if cached:
return json.loads(cached)
response = client.chat.completions.create(model=model, messages=messages, **kwargs)
result = {
"content": response.choices[0].message.content,
"model": response.model,
"tokens": response.usage.prompt_tokens + response.usage.completion_tokens,
}
r.setex(key, ttl, json.dumps(result))
return result
Anthropic models on OpenRouter support prompt caching -- large system prompts are cached server-side, reducing input cost by 90% on cache hits.
# Mark large static content blocks with cache_control
response = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are an expert. Here is the full source:\n" + large_context,
"cache_control": {"type": "ephemeral"}, # Cache this block
}
],
},
{"role": "user", "content": "What does the main() function do?"},
],
max_tokens=1024,
)
# First call: cache_creation_input_tokens charged at 1.25x
# Subsequent: cache_read_input_tokens charged at 0.1x (90% savings)
def cache_key(model: str, messages: list, **params) -> str:
"""Deterministic cache key. Include everything that affects output.
Include: model ID (with variant like :floor), messages, temperature,
max_tokens, top_p, transforms, provider routing.
Exclude: stream (doesn't affect content), HTTP-Referer, X-Title.
"""
canonical = json.dumps({
"model": model, "messages": messages,
"temperature": params.get("temperature", 0),
"max_tokens": params.get("max_tokens"),
"top_p": params.get("top_p"),
}, sort_keys=True)
return hashlib.sha256(canonical.encode()).hexdigest()
| Trigger | Action | Why |
|---|---|---|
| Model version update | Flush keys for that model | New version may give different outputs |
| System prompt change | Flush all keys | Output semantics changed |
| TTL expiry | Automatic eviction | Prevents stale data |
| Manual purge | r.delete(key) or clear by prefix | Debugging or policy change |
{"content", "model", "usage"} from the in-memory cache or {"content", "model", "tokens"} from Redisor:<sha256-of-canonical-request> that expire automatically via TTLhit_rate figure you can use to justify the caching infrastructurecache_creation_input_tokens billed at 1.25x on the first call and cache_read_input_tokens at 0.1x (90% savings) on subsequent hitsTwo identical deterministic calls through the ResponseCache from the references — the second returns instantly from cache:
result1 = cached_completion("What is Python?") # [Cache MISS] key=3f8a92c1... (stored)
result2 = cached_completion("What is Python?") # [Cache HIT] key=3f8a92c1...
print(f"Hit rate: {cache.hit_rate:.0%}") # Hit rate: 50%
More worked examples, including a TypeScript Redis-style cache: references/examples.md.
| Error | Cause | Fix |
|---|---|---|
| Stale cache response | TTL too long | Reduce TTL or version cache keys |
| Cache miss storm | Cold start or invalidation | Warm cache with common queries at deploy |
| Redis connection error | Redis down | Fall through to direct API call |
| Non-deterministic cache | temperature > 0 cached | Only cache when temperature=0 |
temperature=0) -- non-zero temperatures produce different outputs each time
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).