Firecrawl Crawl

Initiates a Firecrawl web crawl from a given URL, applying various filtering and content extraction rules, and polls until the job is complete; ensure the URL is accessible and any regex patterns for paths are valid.

Precio

$0.05 / llamada

Pago unificado

Compatible con tu saldo en USD de Aident

• urlstring

  Obligatorio
  The base URL to start crawling from. This is the initial entry point for the web crawler.
• delayinteger
  Delay in milliseconds between requests to avoid overwhelming the server
• limitinteger
  Maximum number of pages to crawl. The crawl will stop once this limit is reached. Default is 10.
• webhookstring
  An optional webhook URL to receive real-time updates on the crawl job. Events include crawl start (`crawl.started`), page crawled (`crawl.page`), and crawl completion (`crawl.completed` or `crawl.failed`). The payload structure matches the `/scrape` endpoint response.
• maxDepthinteger
  Maximum depth of subpages to crawl relative to the entered URL (not the base domain). A depth of 0 crawls only the entered URL, 1 crawls the entered URL plus pages one path segment deeper, 2 adds two segments deeper, etc. For example, if URL is 'https://example.com/docs/api/', m…
• excludePathsarray
  A list of Regular Expression (regex) patterns for URL paths to exclude from the crawl. URLs whose paths match any of these patterns will be ignored. For example, `"blog/archive/.*"` would exclude all paths under `/blog/archive/`.
• includePathsarray
  A list of Regular Expression (regex) patterns for URL paths to include in the crawl. Only URLs whose paths match one of these patterns will be processed. For example, `"products/featured/.*"` would only include paths under `/products/featured/`.
• ignoreSitemapboolean
  If true, the crawler will ignore any sitemap.xml found on the website.
• crawlEntireDomainboolean
  If true, allows the crawler to follow internal links to sibling or parent URLs, not just child paths. This is the recommended replacement for 'allowBackwardLinks'.
• maxDiscoveryDepthinteger
  Maximum depth to crawl based on discovery order. The root site and sitemapped pages have a discovery depth of 0. For example, if you set it to 1 and set ignoreSitemap, you will only crawl the entered URL and all URLs that are linked on that page.
• allowBackwardLinksboolean
  DEPRECATED: Use 'crawlEntireDomain' instead. If true, allows the crawler to navigate to pages that were linked from pages already visited (i.e., navigate 'backwards').
• allowExternalLinksboolean
  If true, allows the crawler to follow links that lead to external websites (different domains).

• data

  Obligatorio
  Data from the action execution
• errorstring
  Error if any occurred during the execution of the action
• successfulboolean

  Obligatorio
  Whether or not the action execution was successful or not

Aún no hay casos de uso relacionados verificados.