BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions14

Web Scrapper → Monitor Progress Snapshot

AI-generated

Overview

This node interacts with the Bright Data platform to manage web scraping snapshots and monitor their progress. Specifically, the 'Monitor Progress Snapshot' operation allows users to check the progress status of a given snapshot by its ID. This is useful for workflows that need to track the completion or current state of data scraping tasks. For example, a user can automate monitoring of snapshot progress to trigger subsequent actions once data scraping is complete.

Use Case Examples

  1. Monitor the progress of a web scraping snapshot by providing its Snapshot ID to check if the data collection is finished or still ongoing.
  2. Use the snapshot progress information to conditionally trigger data processing or notifications in an automated workflow.

Properties

Name Meaning
Snapshot ID The unique identifier of the snapshot to monitor progress for. This is a required input to specify which snapshot's progress is being queried.

Output

JSON

  • progress - The progress status information of the specified snapshot, as returned by the Bright Data API.

Dependencies

  • Requires an API key credential for Bright Data to authenticate requests to the Bright Data API endpoints.

Troubleshooting

  • If the Snapshot ID is missing or invalid, the node throws an error indicating 'Snapshot ID is required'. Ensure the Snapshot ID is correctly provided.
  • API request failures or network issues will result in errors from the Bright Data API call. Check API credentials and network connectivity.
  • Invalid or expired Snapshot IDs may cause the API to return errors. Verify the snapshot exists and is accessible.

Links

Discussion