Snap Kit icon

Snap Kit

Consume Snap Kit API

Actions2

Crawl Content

AI-generated

Overview

This node interacts with the Snap Kit API to perform web content crawling. Specifically, the 'Crawl Content' operation sends a request to capture and extract content from a specified URL. It supports synchronous or asynchronous execution and allows setting a timeout for the operation. This node is useful for scenarios where automated content extraction from web pages is needed, such as data scraping, monitoring website changes, or archiving web content.

Use Case Examples

  1. Extract content from a news article URL to analyze the text.
  2. Crawl product pages on an e-commerce site to gather pricing and description data.
  3. Monitor changes on a blog by periodically crawling its content.

Properties

Name Meaning
URL The URL to capture (must be a valid URI)
Operation Timeout Timeout in seconds (10-120) for the crawling operation
Sync Whether to wait for capture completion before responding (true for synchronous, false for asynchronous)

Output

JSON

  • url - The URL that was crawled
  • content - The extracted content from the crawled URL
  • status - Status of the crawl operation, e.g., success or failure
  • error - Error message if the crawl operation failed

Dependencies

  • Snap Kit API accessed via an API key credential

Troubleshooting

  • Common issues include invalid URL format, network connectivity problems, or API request timeouts.
  • If the node throws an error related to timeout, consider increasing the 'Operation Timeout' value within the allowed range (10-120 seconds).
  • Ensure the API key credential is correctly configured and has the necessary permissions to access the Snap Kit API.

Discussion