WaterCrawl icon

WaterCrawl

Consume WaterCrawl API

Actions6

Overview

The node integrates with the WaterCrawl API to manage web crawling and scraping tasks. Specifically, the "Stop" operation under the "Crawl" resource allows users to stop an ongoing crawl request by providing its unique Crawl ID. This is useful when you want to halt a crawl that is no longer needed or if it was started by mistake.

Practical scenarios include:

  • Stopping a long-running or stuck crawl job to save resources.
  • Canceling a crawl after realizing the parameters were incorrect.
  • Managing crawl workflows dynamically based on external conditions.

Properties

Name Meaning
Crawl ID The unique identifier of the crawl request to stop.

Output

The output JSON contains a confirmation of the stop action with two fields:

  • success: A boolean indicating whether the stop request was successful (true).
  • message: A string message confirming the crawl request has been stopped, including the Crawl ID.

Example output JSON:

{
  "success": true,
  "message": "Crawl request <Crawl ID> stopped successfully"
}

No binary data is produced by this operation.

Dependencies

  • Requires an API key credential for authenticating with the WaterCrawl API.
  • The base URL for the API defaults to https://app.watercrawl.dev but can be overridden in credentials.
  • The node depends on the WaterCrawlAPIClient class to interact with the WaterCrawl service.

Troubleshooting

  • Invalid Crawl ID: If the provided Crawl ID does not exist or is malformed, the API may return an error. Ensure the Crawl ID is correct.
  • API Authentication Errors: Missing or invalid API key will cause authentication failures. Verify the API key credential is correctly configured.
  • Network Issues: Connectivity problems to the WaterCrawl API endpoint can cause timeouts or errors.
  • Continue On Fail: If enabled, errors during stopping a crawl will be returned as error messages in the output instead of failing the entire workflow.

Common error messages:

  • "Invalid JSON in ..." — Not applicable for Stop operation since it only requires a string Crawl ID.
  • API errors related to authorization or resource not found should be checked against the WaterCrawl API documentation.

Links and References

Discussion