Execute Groq incident response procedures with triage, mitigation, and postmortem.
Use when responding to Groq-related outages, investigating errors,
or running post-incident reviews for Groq integration failures.
Trigger with phrases like "groq incident", "groq outage",
"groq down", "groq on-call", "groq emergency", "groq broken".
Rapid incident response procedures for Groq API failures. Groq is a third-party inference provider -- when it goes down, your mitigation options are: wait, fall back to a different model, or fall back to a different provider.
Is the Groq API responding?
├─ NO (timeout/connection refused):
│ ├─ Check status.groq.com
│ │ ├─ Incident reported → Wait, enable fallback provider
│ │ └─ No incident → Network issue on our side (check DNS, firewall, proxy)
│ └─ Check if api.groq.com resolves: dig api.groq.com
│
├─ YES, but 401/403:
│ ├─ API key revoked or expired → Rotate key
│ └─ Key not set in environment → Check secret manager
│
├─ YES, but 429:
│ ├─ retry-after header present → Wait that many seconds
│ ├─ All models 429 → Org-level limit hit; reduce traffic or upgrade plan
│ └─ One model 429 → Route to a different model
│
├─ YES, but 500/503:
│ ├─ One model → Groq capacity issue on that model; use fallback model
│ └─ All models → Groq-wide outage; enable fallback provider
│
└─ YES, but slow (latency > 2s):
├─ Large prompts → Reduce input size
├─ 70B model → Switch to 8B for speed
└─ queue_time high → Groq queue congestion; try different model
Immediate Mitigations
Enable Fallback to Different Model
// If primary model is failing, route to fallback
async function mitigateModelFailure(messages: any[]) {
const models = [
"llama-3.3-70b-versatile", // Primary
"llama-3.3-70b-specdec", // Same quality, different infra
"llama-3.1-8b-instant", // Fastest, most available
];
for (const model of models) {
try {
return await groq.chat.completions.create({
model,
messages,
max_tokens: 1024,
timeout: 10_000,
});
} catch (err: any) {
console.warn(`Model ${model} failed: ${err.status} ${err.message}`);
continue;
}
}
throw new Error("All Groq models unavailable");
}
429 Rate Limit — Immediate Actions
set -euo pipefail
# Check exact limit info
curl -si https://api.groq.com/openai/v1/chat/completions \
-H "Authorization: Bearer $GROQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"llama-3.1-8b-instant","messages":[{"role":"user","content":"ping"}],"max_tokens":1}' \
2>/dev/null | grep -i "x-ratelimit\|retry-after"
# Options:
# 1. Wait for retry-after seconds
# 2. Switch to a different model (each model has separate limits)
# 3. Reduce request volume (disable non-critical features)
# 4. If persistent, upgrade Groq plan at console.groq.com
401 Auth Failure — Key Rotation
set -euo pipefail
# 1. Verify current key
echo "Current key prefix: ${GROQ_API_KEY:0:8}"
# 2. Create new key at console.groq.com/keys
# 3. Test new key
curl -s -o /dev/null -w "%{http_code}" \
https://api.groq.com/openai/v1/models \
-H "Authorization: Bearer $NEW_GROQ_KEY"
# 4. Deploy new key to production
# 5. Delete old key in console
Communication Templates
Internal Alert (Slack/PagerDuty)
P[1-4] INCIDENT: Groq API [Error Type]
Status: INVESTIGATING | MITIGATING | RESOLVED
Impact: [What users see]
Current action: [What we're doing]
Fallback: [Enabled/Disabled]
Next update in: [Time]
Commander: @[name]
Status Page (External)
AI Feature Performance Issue
We're experiencing [degraded performance / intermittent errors] with our AI features.
[Feature X] may respond slower than usual.
We've activated backup systems and are monitoring the situation.
Last updated: [timestamp]
