Optimize FireCrawl API performance with caching, batching, and connection pooling.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for FireCrawl integrations.
Trigger with phrases like "firecrawl performance", "optimize firecrawl",
"firecrawl latency", "firecrawl caching", "firecrawl slow", "firecrawl batch".
Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), waitFor tuning, onlyMainContent, and batch vs individual scraping.
Latency Benchmarks
Operation
Typical
With JS Wait
With Screenshot
scrapeUrl (markdown)
2-5s
5-10s
8-15s
scrapeUrl (extract)
3-8s
8-15s
N/A
crawlUrl (10 pages)
20-40s
40-80s
N/A
mapUrl
1-3s
N/A
N/A
batchScrapeUrls (10)
10-20s
20-40s
N/A
Instructions
Step 1: Minimize Formats (Biggest Win)
import FirecrawlApp from "@mendable/firecrawl-js";
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
// SLOW: requesting everything
const slow = await firecrawl.scrapeUrl(url, {
formats: ["markdown", "html", "links", "screenshot"],
// screenshot + full HTML = 3-5x slower
});
// FAST: request only what you need
const fast = await firecrawl.scrapeUrl(url, {
formats: ["markdown"], // markdown only = fastest
onlyMainContent: true, // skip nav/footer/sidebar
});
Step 2: Tune waitFor for JS-Heavy Pages
// Default: no JS wait (fastest, works for static sites)
const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", {
formats: ["markdown"],
// No waitFor needed — content is in initial HTML
});
// SPA/dynamic pages: add minimal wait
const spaResult = await firecrawl.scrapeUrl("https://app.example.com", {
formats: ["markdown"],
waitFor: 3000, // 3s — enough for most SPAs
onlyMainContent: true,
});
// Heavy interactive page: use actions instead of long wait
const heavyResult = await firecrawl.scrapeUrl("https://dashboard.example.com", {
formats: ["markdown"],
actions: [
{ type: "wait", selector: ".data-table" }, // wait for specific element
{ type: "scroll", direction: "down" }, // trigger lazy loading
],
});
Step 3: Cache Scraped Content
import { LRUCache } from "lru-cache";
import { createHash } from "crypto";
const scrapeCache = new LRUCache<string, any>({
max: 500, // max 500 cached pages
ttl: 3600000, // 1 hour TTL
});
async function cachedScrape(url: string) {
const key = createHash("md5").update(url).digest("hex");
const cached = scrapeCache.get(key);
if (cached) {
console.log(`Cache hit: ${url}`);
return cached;
}
const result = await firecrawl.scrapeUrl(url, {
formats: ["markdown"],
onlyMainContent: true,
});
if (result.success) {
scrapeCache.set(key, result);
}
return result;
}
// Typical savings: 50-80% credit reduction for repeated scrapes
