Firecrawl Crawl V2

The base URL to start crawling from. This is the initial entry point for the web crawler.

Delay in seconds between scrapes to respect website rate limits.

Maximum number of pages to crawl. The crawl will stop once this limit is reached. Default is 10.

A natural language prompt to automatically generate crawler settings. Example: 'Only crawl blog posts and docs, skip marketing pages'. Explicitly set parameters will override the generated equivalents.

Sitemap mode when crawling. 'include' (default) uses sitemap and discovers other pages. 'skip' ignores sitemap entirely and only discovers pages from the start URL. 'only' crawls exclusively URLs from the sitemap, ignoring other discovered links.

Webhook configuration for receiving real-time crawl updates.

A list of Regular Expression (regex) patterns for URL paths to exclude from the crawl. URLs whose paths match any of these patterns will be ignored. For example, `"blog/archive/.*"` would exclude all paths under `/blog/archive/`.

A list of Regular Expression (regex) patterns for URL paths to include in the crawl. Only URLs whose paths match one of these patterns will be processed. For example, `"products/featured/.*"` would only include paths under `/products/featured/`.

Maximum number of concurrent scrapes. If not specified, uses your team's concurrency limit.

If true, allows the crawler to follow links to subdomains of the main domain.

Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. False: Only crawls deeper (child) URLs (e.g., /features/feature-1 → /features/feature-1/tips). True: Crawls any internal links including siblings and parents (e.g., /features/feature-1 →…

Maximum depth to crawl based on discovery order. The root site and sitemapped pages have a discovery depth of 0. For example, if you set it to 1 and set sitemap='skip', you will only crawl the entered URL and all URLs linked on that page.