BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions14

Web Scrapper → Trigger Collection By URL

AI-generated

Overview

This node interacts with the Bright Data platform to trigger web scraping collections by URL using a specified dataset. It allows users to initiate data snapshots from given URLs, specifying where to send the scraped data and where to notify upon completion. This is useful for automating data collection from web sources, such as monitoring competitor websites, gathering market data, or extracting public profile information from URLs.

Use Case Examples

  1. Trigger a web scraping collection for a list of URLs using a selected dataset, sending the scraped data to a specified endpoint and receiving a notification when the collection finishes.
  2. Use the node to automate data extraction from multiple URLs for market research or data aggregation tasks.

Properties

Name Meaning
Dataset Select the dataset to be used for the web scraping collection. This is required to identify the data source configuration on Bright Data.
URLs A JSON array of URLs to trigger the snapshot collection. This input defines the web pages to be scraped.
Endpoint The endpoint URL where the data obtained from the snapshot will be sent.
Notify The URL to notify when the collection is finished.

Output

JSON

  • dataset_id - The identifier of the dataset used for the collection.
  • endpoint - The endpoint URL where the scraped data is sent.
  • notify - The notification URL called when the collection finishes.
  • response - The response from the Bright Data API after triggering the collection.

Dependencies

  • Requires an API key credential for Bright Data platform authentication to make API requests.

Troubleshooting

  • Invalid JSON format for URLs input will cause an error; ensure the URLs are provided as a valid JSON array.
  • Dataset ID is required; if missing, the node will throw an error indicating the necessity of this parameter.
  • API request failures will throw errors; check API credentials and network connectivity.
  • Ensure the endpoint and notify URLs are valid and reachable to avoid delivery or notification failures.

Discussion