Crawl4AI: Basic CrawlerInstall

Summary

Create async crawl job with webhook. Posts crawl request to /crawl/job endpoint and returns job ID immediately. Workflow can later monitor job status via 'Check Job' operation or receive webhook notification on completion.

Inputs

• urls (required) — Comma-separated list of URLs to crawl. Example: 'https://example.com, https://example2.com'. Split, trimmed, and filtered internally.
• outputFormat — Format of data to extract from crawled pages. Default: markdown. Options: markdown (clean markdown content), html (HTML content), full (complete CrawlResult object).
• webhookConfig — Configuration for webhook notification on job completion. Only available for Async Job operation. When webhookUrl is set, the API receives webhook_config in the request body. Includes webhookUrl (required for webhook), webhookDataInPayload (default true), and webhookHeaders (optional JSON).
• browserConfig — Browser launch and execution settings. Key options: headless (boolean, default true), enableStealth (boolean, default false), userAgent (string), viewportWidth (number, default 1920), viewportHeight (number, default 1080).
• crawlerConfig — Crawl execution parameters. Key options: cacheMode (options: bypass/enabled/write_only/read_only, default bypass), pageTimeout (number in seconds, default 30, multiplied by 1000 for API), cssSelector (string), jsCode (string), screenshot (boolean, default false), removeOverlays (boolean, default true), excludeExternalLinks (boolean, default false), waitFor (string).
• sessionConfig — Session management configuration. Key options: sessionId (string), cookiesJson (string, JSON array format), cookiesNetscape (string, Netscape format). Cannot use both JSON and Netscape cookies simultaneously.
• proxyConfig — Proxy server configuration. Key options: server (string, e.g., 'http://proxy.example.com:8080'), username (string), password (string).
• additionalOptions — Miscellaneous crawl options. Key options: customHeaders (string, JSON format), delayBeforeReturn (number seconds), magic (boolean, default false), simulateUser (boolean, default false), stream (boolean, default false), waitTime (number seconds).

Output shape

job_object

Returns the raw API response object containing job identification and status information. The workflow can extract the job ID to later monitor progress via the 'Check Job' operation (operation: checkJob, requiring taskId parameter), or wait for webhook delivery if webhookConfig was configured with a webhookUrl.

Examples

Example 1: Basic async crawl with webhook notification

Set urls to target URLs, configure webhookConfig with a webhookUrl endpoint, and the node will return immediately with a job ID. The external webhook endpoint will receive a notification when the crawl completes.

Example 2: Monitor async job status in subsequent workflow steps

Capture the job ID from the Async Job output, then use the Check Job operation with the taskId parameter to poll job status and retrieve results once complete.