Implement FireCrawl rate limiting, backoff, and idempotency patterns.
Use when handling rate limit errors, implementing retry logic,
or optimizing API request throughput for FireCrawl.
Trigger with phrases like "firecrawl rate limit", "firecrawl throttling",
"firecrawl 429", "firecrawl retry", "firecrawl backoff".
Firecrawl enforces rate limits per API key measured in requests per minute and concurrent connections. When exceeded, the API returns 429 Too Many Requests with a Retry-After header. This skill covers backoff strategies, request queuing, and proactive throttling.
Rate Limit Tiers
Plan
Scrape RPM
Crawl Concurrency
Credits/Month
Free
10
2
500
Hobby
20
3
3,000
Standard
50
5
50,000
Growth
100
10
500,000
Scale
500+
50+
Custom
Concurrent crawl jobs count against concurrency limits. If the queue is full, new jobs are rejected with 429.
Instructions
Step 1: Exponential Backoff with Jitter
import FirecrawlApp from "@mendable/firecrawl-js";
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
async function withBackoff<T>(
operation: () => Promise<T>,
config = { maxRetries: 5, baseDelayMs: 1000, maxDelayMs: 32000 }
): Promise<T> {
for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
try {
return await operation();
} catch (error: any) {
if (attempt === config.maxRetries) throw error;
const status = error.statusCode || error.status;
// Only retry on 429 (rate limit) and 5xx (server error)
if (status && status !== 429 && status < 500) throw error;
// Exponential delay with random jitter to prevent thundering herd
const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);
const jitter = Math.random() * 500;
const delay = Math.min(exponentialDelay + jitter, config.maxDelayMs);
console.warn(`Rate limited (${status}). Retry ${attempt + 1}/${config.maxRetries} in ${delay.toFixed(0)}ms`);
await new Promise(r => setTimeout(r, delay));
}
}
throw new Error("Unreachable");
}
// Usage
const result = await withBackoff(() =>
firecrawl.scrapeUrl("https://example.com", { formats: ["markdown"] })
);
Step 2: Queue-Based Rate Limiting with p-queue
import PQueue from "p-queue";
// Limit to 5 concurrent requests, max 10 per second
const scrapeQueue = new PQueue({
concurrency: 5,
interval: 1000,
intervalCap: 10,
});
async function queuedScrape(url: string) {
return scrapeQueue.add(() =>
withBackoff(() =>
firecrawl.scrapeUrl(url, { formats: ["markdown"] })
)
);
}
// Scrape many URLs respecting rate limits
const urls = ["https://a.com", "https://b.com", "https://c.com"];
const results = await Promise.all(urls.map(url => queuedScrape(url)));
console.log(`Queue: ${scrapeQueue.pending} pending, ${scrapeQueue.size} queued`);
Step 3: Proactive Throttling (Pre-emptive)
class RateLimitTracker {
private requestTimes: number[] = [];
private windowMs: number;
private maxRequests: number;
constructor(maxRequests = 50, windowMs = 60000) {
this.maxRequests = maxRequests;
this.windowMs = windowMs;
}
async waitIfNeeded(): Promise<void> {
const now = Date.now();
this.requestTimes = this.requestTimes.filter(t => now - t < this.windowMs);
if (this.requestTimes.length >= this.maxRequests) {
const oldestInWindow = this.requestTimes[0];
const waitMs = this.windowMs - (now - oldestInWindow) + 100;
console.log(`Proactive throttle: waiting ${waitMs}ms to stay under ${this.maxRequests} RPM`);
await new Promise(r => setTimeout(r, waitMs));
}
this.requestTimes.push(Date.now());
}
}
const throttle = new RateLimitTracker(50, 60000); // 50 requests per minute
async function throttledScrape(url: string) {
await throttle.waitIfNeeded();
return firecrawl.scrapeUrl(url, { formats: ["markdown"] });
}
Step 4: Batch Scrape for Efficiency
// batchScrapeUrls is more efficient than individual scrapes
// It handles internal rate limiting and is cheaper on credits
const urls = [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3",
];
// Single API call instead of 3 separate scrapes
const batchResult = await firecrawl.batchScrapeUrls(urls, {
formats: ["markdown"],
});
console.log(`Batch scraped ${batchResult.data?.length} pages`);
Error Handling
Header
Description
Action
Retry-After
Seconds to wait
Honor this exact value
X-RateLimit-Limit
Max requests per window
Use for proactive throttling
X-RateLimit-Remaining
Remaining in window
Slow down when < 5
X-RateLimit-Reset
Reset timestamp
Wait until this time
Examples
Monitor Rate Limit Usage
class RateLimitMonitor {
private remaining = Infinity;
private resetAt = new Date();
update(status: number, headers: Record<string, string>) {
if (headers["x-ratelimit-remaining"]) {
this.remaining = parseInt(headers["x-ratelimit-remaining"]);
}
if (headers["x-ratelimit-reset"]) {
this.resetAt = new Date(parseInt(headers["x-ratelimit-reset"]) * 1000);
}
if (this.remaining < 5) {
console.warn(`Low rate limit: ${this.remaining} remaining, resets at ${this.resetAt.toISOString()}`);
}
}
}
