BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions14

Web Scrapper → Get Snapshots

AI-generated

Overview

This node interacts with the Bright Data platform to retrieve snapshots of web scraping datasets. Specifically, the 'Get Snapshots' operation fetches snapshots filtered by dataset ID, status, date range, and pagination parameters. It is useful for users who want to monitor or analyze the collected data snapshots from their web scraping activities. For example, a user can retrieve all snapshots marked as 'ready' within a specific date range for a particular dataset to process or analyze the scraped data.

Use Case Examples

  1. Retrieve all 'ready' snapshots for a selected dataset between two dates.
  2. Fetch snapshots with a specific status like 'failed' to investigate scraping issues.
  3. Paginate through snapshots by skipping a number of entries and limiting the results.

Properties

Name Meaning
Dataset The dataset from which to retrieve snapshots. This is a required resource locator input allowing selection from a list of datasets.
Status Filter snapshots by their status, such as 'ready', 'failed', or 'collecting'. Defaults to 'ready' if not specified.
Skip Number of snapshots to skip in the result set, useful for pagination.
Limit Maximum number of snapshot results to return, with a minimum value of 1.
From Date Start date (ISO 8601 format) to filter snapshots. Required to define the beginning of the date range.
To Date End date (ISO 8601 format) to filter snapshots. Required to define the end of the date range.

Output

JSON

  • dataset_id - Identifier of the dataset the snapshots belong to.
  • snapshots - Array of snapshot objects matching the filter criteria, each containing metadata and status information.
  • status - Status of each snapshot, indicating its current state in the processing pipeline.
  • created_at - Timestamp when the snapshot was created.
  • updated_at - Timestamp when the snapshot was last updated.

Dependencies

  • Requires an API key credential for Bright Data platform authentication to make authorized requests.

Troubleshooting

  • Common issues include missing or invalid dataset ID, which will cause the node to throw an error indicating the requirement.
  • Invalid date formats for 'From Date' or 'To Date' can cause request failures; ensure dates are in ISO 8601 format.
  • API request failures due to network issues or invalid credentials will result in errors; verify API key and network connectivity.

Links

Discussion