Crawl4AI Plus Advanced icon

Crawl4AI Plus Advanced

Advanced web crawling and extraction with full Crawl4AI API control

Actions15

Submit Crawl Job

AI-generated

Summary

Submit a crawl job for one or more URLs to the Crawl4AI Plus Advanced service, configuring browser sessions, crawl settings, and optional webhook notifications for asynchronous crawling.

Inputs

  • URLs (required) — Provide one or more URLs to crawl, specified as a newline-separated string.
  • Browser & Session — Configure the browser environment including browser engine (Chromium, Firefox, WebKit), cookies, JavaScript execution, stealth mode, viewport size, proxy settings, user agent, headless mode, and session persistence options.
  • Crawl Settings — Specify crawl behavior such as anti-bot measures (magic mode, navigator override, user simulation), cache usage mode, respect for robots.txt, CSS selector to limit extraction, delay after page load before returning HTML, external link exclusion, excluded HTML tags, JavaScript execution on the page, retry attempts, wait conditions, and content word count threshold.
  • Webhook Config — Define a webhook URL and optional custom headers to receive POST notifications when the crawl job completes, with option to include crawl result data in the webhook payload.

Output shape

a single record containing the response with the job/task ID from submitting the asynchronous crawl job to Crawl4AI Plus Advanced.

The operation submits the job asynchronously and returns immediately with a task identifier. The actual crawl results are delivered later via the configured webhook or can be retrieved using a separate job status operation.

Links

Discussion