Troubleshoot and respond to Langfuse-related incidents and outages.
Use when experiencing Langfuse outages, debugging production issues,
or responding to LLM observability incidents.
Trigger with phrases like "langfuse incident", "langfuse outage",
"langfuse down", "langfuse production issue", "langfuse troubleshoot".
Step-by-step procedures for Langfuse-related incidents, from initial triage (2 min) through resolution and post-incident review. Your application should work without Langfuse -- these procedures focus on restoring observability.
Check shutdown handlers; set flushAt: 1 temporarily
401 Unauthorized
Key rotation or mismatch
Verify keys match the correct project
429 Too Many Requests
Rate limited
Increase batch size, reduce flush frequency
SDK throwing errors
Unhandled exception
Wrap in try/catch; check SDK version
High request latency
Sync flush in hot path
Switch to async; increase requestTimeout
Complete Langfuse outage
Service-side issue
Enable fallback mode
Step 3: Fallback Mode (P1 -- App Impacted)
If Langfuse is causing application issues, disable tracing immediately:
// Emergency disable via environment variable
// Set LANGFUSE_ENABLED=false in your deployment
// In your tracing initialization:
if (process.env.LANGFUSE_ENABLED === "false") {
console.warn("Langfuse tracing DISABLED (emergency fallback)");
// Don't initialize SDK -- all observe/startActiveObservation calls
// will still work but produce no-op spans
}
For v3, use the enabled flag:
const langfuse = new Langfuse({
enabled: process.env.LANGFUSE_ENABLED !== "false",
});
Step 4: Common Resolution Procedures
Procedure A: Missing Traces
// 1. Verify SDK is initialized
console.log("Langfuse configured:", !!process.env.LANGFUSE_PUBLIC_KEY);
// 2. Check flush is happening
// v4+: Verify NodeSDK is started and shutdown is registered
// v3: Verify flushAsync() or shutdownAsync() is called
// 3. Temporarily set aggressive flush for debugging
const processor = new LangfuseSpanProcessor({
exportIntervalMillis: 1000,
maxExportBatchSize: 1,
});
Procedure B: Rate Limit (429) Recovery
// Increase batching to reduce API calls
const processor = new LangfuseSpanProcessor({
exportIntervalMillis: 30000, // 30s flush
maxExportBatchSize: 200, // Large batches
});
// Or temporarily enable sampling
const EMERGENCY_SAMPLE_RATE = 0.1; // Only trace 10%
Procedure C: Self-Hosted Instance Down
set -euo pipefail
# Check container status
docker ps -a | grep langfuse
# Check logs
docker logs langfuse-langfuse-1 --tail 50
# Check database
docker exec langfuse-postgres-1 pg_isready -U langfuse
# Restart if needed
docker compose restart langfuse
Step 5: Post-Incident Verification
set -euo pipefail
# Verify traces are flowing again
echo "=== Post-Incident Check ==="
HOST="${LANGFUSE_BASE_URL:-https://cloud.langfuse.com}"
AUTH=$(echo -n "$LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY" | base64)
# Check recent trace count
TRACE_COUNT=$(curl -s \
-H "Authorization: Basic $AUTH" \
"$HOST/api/public/traces?limit=5" | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('data',[])))" 2>/dev/null || echo "ERROR")
echo "Recent traces: $TRACE_COUNT"
if [ "$TRACE_COUNT" = "0" ] || [ "$TRACE_COUNT" = "ERROR" ]; then
echo "WARNING: Traces may not be flowing yet"
else
echo "OK: Traces are appearing"
fi
Step 6: Post-Incident Review (P1/P2)
Document for post-mortem:
Timeline: When detected, when resolved, total duration
Impact: Traces lost, application impact, user impact
