Diffbot Start Crawl

• namestring

  Required
  Unique identifier for the crawl job. Used to manage and retrieve the crawl.
• seedsarray

  Required
  List of seed URLs where crawling will begin. URLs will be URL-encoded automatically.
• apiUrlstring

  Required
  Full Diffbot Extract API endpoint URL to process crawled pages. Examples: 'https://api.diffbot.com/v3/article' for articles, 'https://api.diffbot.com/v3/product' for products, 'https://api.diffbot.com/v3/analyze' for automatic type detection.
• repeatnumber
  Number of days between automatic crawl repeats. Use 7.0 for weekly, 1.0 for daily. Omit for one-time crawl.
• crawlDelaynumber
  Delay in seconds between requests to the same IP address. Default is 0.25 seconds.
• maxToCrawlinteger
  Maximum number of pages to crawl/spider. Default is 100,000. Use -1 for unlimited.
• obeyRobotsinteger
  Whether to respect robots.txt directives. 1 = obey (default), 0 = ignore.
• notifyEmailstring
  Email address to notify when the crawl completes.
• maxToProcessinteger
  Maximum number of pages to process with the Extract API. Default is 100,000. Use -1 for unlimited.
• customHeadersobject
  Custom HTTP headers to include in crawl requests (e.g., {'User-Agent': 'MyBot/1.0'}).
• notifyWebhookstring
  Webhook URL to POST to when the crawl completes.

• data

  Required
  Data from the action execution
• errorstring
  Error if any occurred during the execution of the action
• successfulboolean

  Required
  Whether or not the action execution was successful or not

No verified related use cases are linked yet.

No published Skills explicitly reference this Action yet.