BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions16

Web Scraper → Trigger Collection By URL

AI-generated

Overview

This node triggers a web scraping collection by URL using the Bright Data service. It allows users to specify a dataset and a list of URLs to initiate snapshot collections. The node sends the scraped data to a specified endpoint and can notify a URL when the collection is finished. This is useful for automating data extraction from multiple web pages and integrating the results into other workflows or systems.

Use Case Examples

  1. Triggering a snapshot collection for a list of LinkedIn profile URLs to gather public profile data.
  2. Automating data extraction from e-commerce product pages by specifying product URLs and sending the data to a custom endpoint for analysis.

Properties

Name Meaning
Dataset Select the dataset to be used for the web scraping collection.
URLs The URLs to trigger the snapshot collection for.
Endpoint The endpoint URL to send the data obtained from the snapshot.
Notify The URL to notify when the collection is finished.
Request Options Additional request options such as batching, SSL certificate handling, proxy settings, and timeout configuration.

Output

JSON

  • data - The data obtained from the snapshot collection.
  • status - The status of the collection trigger operation.
  • message - Additional message or information about the operation result.

Dependencies

  • Bright Data API key credential

Troubleshooting

  • Ensure the dataset ID is valid and accessible with the provided credentials.
  • Verify that the URLs are correctly formatted and reachable.
  • Check the endpoint and notify URLs for correctness and availability.
  • If SSL issues occur, consider enabling the option to ignore SSL certificate validation.
  • Proxy settings must be correctly configured if used, including authentication details.

Links

Discussion