BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions14

Web Scrapper → Scrape By URL

AI-generated

Overview

This node integrates with Bright Data's web scraping services, specifically supporting the 'Web Scrapper' resource with the 'Scrape By URL' operation. It allows users to trigger web scraping tasks by providing a dataset and a list of URLs to scrape. The node sends these URLs to Bright Data's API, specifying the desired data format (JSON or CSV) and whether to include errors in the response. It is useful for automating data extraction from multiple web pages, such as scraping profiles, product details, or other web content, and retrieving the results in a structured format.

Use Case Examples

  1. Scrape product information from a list of e-commerce URLs using a predefined dataset.
  2. Extract LinkedIn profile data by providing profile URLs and receiving structured JSON data.
  3. Collect news articles from multiple URLs and get the data in CSV format for analysis.

Properties

Name Meaning
Dataset The dataset to use for scraping, selected from a list of available datasets.
URLs A JSON array of URLs to be scraped by the dataset.
Include Errors Whether to include errors in the scraping response.
Format The format of the returned data, either JSON or CSV.

Output

JSON

  • data - The scraped data returned from the Bright Data API, either as an array of objects or a string depending on the response.

Dependencies

  • Requires an API key credential for Bright Data to authenticate requests to the Bright Data API.

Troubleshooting

  • Common issues include invalid JSON format for the URLs input, which will cause a parsing error. Ensure the URLs are provided as a valid JSON array.
  • Missing or undefined dataset ID will cause the node to throw an error; always provide a valid dataset selection.
  • API request failures may occur due to network issues or invalid credentials; verify the Bright Data API key and network connectivity.
  • Errors in the response can be included or excluded based on the 'Include Errors' property; enabling this can help diagnose scraping issues.

Links

Discussion