Actions16
- Marketplace Dataset Actions
- Web Scraper Actions
- Web Unlocker Actions
Web Scraper → Deliver Snapshot
AI-generatedOverview
This node operation delivers a snapshot from the Bright Data Web Scraper service to a specified destination. It supports multiple delivery types including cloud storage services (Amazon S3, Aliyun OSS, Google Cloud Storage, Microsoft Azure), messaging services (Google Cloud PubSub), SFTP servers, Snowflake data warehouse, and webhooks. Users can specify the snapshot ID to deliver, configure the delivery method, and provide necessary credentials and parameters for the chosen delivery type. This operation is useful for automating the transfer of scraped data snapshots to various storage or processing endpoints for further use or analysis.
Use Case Examples
- Deliver a web scraper snapshot to an Amazon S3 bucket for storage and later processing.
- Send a snapshot to a Google Cloud PubSub topic to trigger downstream data processing workflows.
- Deliver snapshot data to an SFTP server for integration with legacy systems.
- Use webhook delivery to push snapshot data directly to a custom API endpoint.
Properties
| Name | Meaning |
|---|---|
| Snapshot ID | The ID of the snapshot to operate on, required to identify which snapshot to deliver. |
| Notify | URL where a notification will be sent once the delivery is finished, allowing asynchronous status updates. |
| Deliver Type | The delivery method to use for the snapshot, such as S3, Aliyun OSS, Google Cloud Storage, Azure, SFTP, Snowflake, or Webhook. |
| Webhook Endpoint | Webhook URL to deliver the snapshot, required if delivery type is webhook. |
| Filename Template | Template for the filename, including placeholders, used for naming the delivered file. |
| File Extension | Extension for the delivered file, such as JSON, JSONL, or CSV. |
| Topic ID | Google PubSub topic ID, required if delivery type is Google Cloud PubSub. |
| Client Email | Client email for Google Cloud credentials, required for Google Cloud PubSub and Storage delivery types. |
| Private Key | Private key for Google Cloud credentials, required for Google Cloud PubSub and Storage delivery types. |
| Attributes | Attributes to include in the PubSub message, optional JSON object. |
| Container | Name of the Azure container, required for Microsoft Azure delivery. |
| Bucket | Name of the bucket, required for Amazon S3, Aliyun OSS, and Google Cloud Storage delivery. |
| AWS Access Key | AWS Access Key ID, required for Amazon S3 delivery. |
| AWS Secret Key | AWS Secret Access Key, required for Amazon S3 delivery. |
| Access Key | Access key for Aliyun OSS delivery. |
| Secret Key | Secret key for Aliyun OSS delivery. |
| Account | Azure storage account, required for Microsoft Azure delivery. |
| Key | Azure storage key, required for Microsoft Azure delivery. |
| SAS Token | Azure SAS token for access, required for Microsoft Azure delivery. |
| Role ARN | Role ARN for Amazon S3 delivery, optional for assuming roles. |
| External ID | External ID for Amazon S3 delivery, optional for assuming roles. |
| Directory | Target path or directory in the storage or delivery destination. |
| Region | AWS region, required for Amazon S3 and Aliyun OSS delivery. |
| Host | SFTP server host, required for SFTP delivery. |
| Port | SFTP server port, default is 22. |
| Path | Remote path on the SFTP server to store the file. |
| Username | SFTP username, required for SFTP delivery. |
| Password | SFTP password, required for SFTP delivery. |
| SSH Key | SSH key for SFTP authentication, required for SFTP delivery. |
| Passphrase | Passphrase for the SSH key, if any. |
| Database | Snowflake database name, required for Snowflake delivery. |
| Schema | Snowflake schema name, required for Snowflake delivery. |
| Stage | Snowflake stage name, required for Snowflake delivery. |
| Role | Snowflake role, required for Snowflake delivery. |
| Warehouse | Snowflake warehouse, required for Snowflake delivery. |
| Account (Snowflake) | Snowflake account, required for Snowflake delivery. |
| User (Snowflake) | Snowflake user, required for Snowflake delivery. |
| Password (Snowflake) | Snowflake password, required for Snowflake delivery. |
| Compress | Whether to compress the response in gzip format, optional. |
Output
JSON
snapshotId- The ID of the snapshot that was delivered.status- The status of the delivery operation.message- Additional message or information about the delivery result.
Dependencies
- Bright Data API key credential
Troubleshooting
- Ensure all required credentials for the chosen delivery type are provided and valid to avoid authentication errors.
- Verify the snapshot ID is correct and exists in the Bright Data system to prevent not found errors.
- Check network connectivity and endpoint URLs, especially for webhook and SFTP delivery types, to avoid connection failures.
- If delivery fails due to permission issues, confirm that the provided credentials have sufficient access rights to the target storage or service.
- For Google Cloud PubSub delivery, ensure the client email and private key are correctly configured and the topic ID exists.
Links
- Bright Data Web Scraper API Documentation - Official documentation for using Bright Data's Web Scraper API including snapshot delivery.