BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions16

Web Scraper → Scrape By URL

AI-generated

Overview

This node interacts with the Bright Data service to scrape web data by providing specific URLs and a dataset. It is useful for extracting structured data from web pages using predefined datasets, which can be beneficial for market research, competitive analysis, or data aggregation tasks. For example, a user can input a list of LinkedIn profile URLs and a dataset to scrape profile information in JSON or CSV format.

Use Case Examples

  1. Scraping product details from e-commerce websites by providing product URLs and a dataset that defines the data structure.
  2. Extracting contact information from business websites by specifying URLs and using a dataset tailored for contact data extraction.

Properties

Name Meaning
Dataset Select the dataset to be used for scraping, which defines the data structure and extraction rules.
URLs The list of URLs to scrape data from, provided in JSON format.
Include Errors Whether to include errors in the response, useful for debugging or handling partial failures.
Format The format of the returned data, either JSON or CSV.
Request Options Additional HTTP request options such as batching, SSL certificate handling, proxy settings, and timeout configuration.

Output

JSON

  • data - The scraped data returned from the Bright Data service, formatted as specified (JSON or CSV).
  • errors - Any errors encountered during the scraping process, included if 'Include Errors' is set to true.

Dependencies

  • Bright Data API key credential for authentication with the Bright Data service.

Troubleshooting

  • Ensure the Bright Data API key credential is correctly configured and has the necessary permissions.
  • Verify that the dataset ID and URLs are valid and accessible.
  • Check network settings if using a proxy or if SSL certificate validation is disabled.
  • Timeout errors may occur if the server response is slow; adjust the timeout setting accordingly.

Links

Discussion