BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions14

Web Scrapper → Download Snapshot

AI-generated

Overview

This node interacts with the Bright Data platform to download snapshot data from web scraping operations. Specifically, the 'Download Snapshot' operation allows users to retrieve parts of a snapshot by specifying the snapshot ID, the desired data format (JSON or CSV), batch size, part number, and whether the data should be compressed using gzip. This is useful for processing large datasets in manageable chunks, enabling efficient data retrieval and handling in workflows.

Use Case Examples

  1. Downloading a large web scraping snapshot in JSON format in batches of 1000 records to process data incrementally.
  2. Retrieving a specific part of a snapshot compressed in gzip format to save bandwidth and storage.
  3. Downloading snapshot data in CSV format for integration with spreadsheet or database applications.

Properties

Name Meaning
Batch Size The number of records to download in each batch.
Part The part number of the snapshot to download.
Snapshot ID The ID of the snapshot to operate on. This is required to identify which snapshot to download.
Format The format of the data to be returned, either JSON or CSV.
Compress Whether to compress the response in gzip format to reduce data size.

Output

JSON

  • items - The downloaded snapshot data batch returned from the Bright Data API.

Dependencies

  • Requires an API key credential for Bright Data platform authentication.

Troubleshooting

  • Ensure the 'Snapshot ID' parameter is provided; otherwise, the node throws an error indicating it is required.
  • If the API request fails, the node throws an error with the API response message; check network connectivity and API credentials.
  • Invalid 'Format' or 'Compress' parameter values may cause unexpected results; use the provided options only.

Links

Discussion