Actions14
- Marketplace Dataset Actions
- Web Scrapper Actions
- Web Unlocker Actions
Web Scrapper → Deliver Snapshot
AI-generatedOverview
This node interacts with the Bright Data platform to deliver snapshots of scraped web data to various storage or delivery endpoints. It supports multiple delivery types including cloud storage services (Amazon S3, Aliyun OSS, Google Cloud Storage, Microsoft Azure), messaging services (Google Cloud PubSub), SFTP servers, Snowflake data warehouse, and webhooks. Users provide a snapshot ID and configure delivery parameters such as filename templates, file extensions, and credentials for the target service. The node sends a delivery request to Bright Data, which then processes and delivers the snapshot accordingly. This is useful for automating the distribution of web scraping results to preferred storage or processing systems.
Use Case Examples
- Deliver a web scraping snapshot to an Amazon S3 bucket with a specified filename template and JSON format.
- Send a snapshot to a Google Cloud PubSub topic with custom message attributes for downstream processing.
- Deliver snapshot data to an SFTP server using SSH key authentication and a CSV file format.
Properties
| Name | Meaning |
|---|---|
| Snapshot ID | The ID of the snapshot to operate on, required to identify which snapshot to deliver. |
| Notify | URL where a notification will be sent once the delivery is finished. |
| Deliver Type | The type of delivery endpoint to use for snapshot delivery, such as S3, webhook, or SFTP. |
| Webhook Endpoint | Webhook URL to deliver the snapshot, required if deliver type is webhook. |
| Filename Template | Template for the filename including placeholders, required for all delivery types. |
| File Extension | File extension for the delivered file, such as JSON, JSONL, or CSV. |
| Topic ID | Google PubSub topic ID, required if deliver type is gcs_pubsub. |
| Client Email | Client email for Google Cloud credentials, required for gcs and gcs_pubsub delivery types. |
| Private Key | Private key for Google Cloud credentials, required for gcs and gcs_pubsub delivery types. |
| Attributes | Attributes to include in the PubSub message, applicable for gcs_pubsub delivery type. |
| Container | Name of the Azure container, required for azure delivery type. |
| Bucket | Name of the bucket, required for s3, ali_oss, and gcs delivery types. |
| AWS Access Key | AWS Access Key ID, required for s3 delivery type. |
| AWS Secret Key | AWS Secret Access Key, required for s3 delivery type. |
| Access Key | Access key for Aliyun OSS, required for ali_oss delivery type. |
| Secret Key | Secret key for Aliyun OSS, required for ali_oss delivery type. |
| Account | Azure storage account, required for azure delivery type. |
| Key | Azure storage key, required for azure delivery type. |
| SAS Token | Azure SAS token for access, required for azure delivery type. |
| Role ARN | AWS role ARN, optional for s3 delivery type. |
| External ID | External ID for AWS role assumption, optional for s3 delivery type. |
| Directory | Target path or directory for storing the file, applicable for multiple delivery types. |
| Region | AWS region, applicable for s3 and ali_oss delivery types. |
| Host | SFTP server host, required for sftp delivery type. |
| Port | SFTP server port, default is 22, applicable for sftp delivery type. |
| Path | Remote path on the SFTP server to store the file, applicable for sftp delivery type. |
| Username | SFTP username, required for sftp delivery type. |
| Password | SFTP password, required for sftp delivery type. |
| SSH Key | SSH key for SFTP authentication, required for sftp delivery type. |
| Passphrase | Passphrase for the SSH key, if any, applicable for sftp delivery type. |
| Database | Snowflake database name, required for snowflake delivery type. |
| Schema | Snowflake schema name, required for snowflake delivery type. |
| Stage | Snowflake stage name, required for snowflake delivery type. |
| Role | Snowflake role, required for snowflake delivery type. |
| Warehouse | Snowflake warehouse, required for snowflake delivery type. |
| Account (Snowflake) | Snowflake account credential, required for snowflake delivery type. |
| User (Snowflake) | Snowflake user credential, required for snowflake delivery type. |
| Password (Snowflake) | Snowflake password credential, required for snowflake delivery type. |
| Compress | Whether to compress the response in gzip format, applicable for deliverSnapshot and downloadSnapshot operations. |
Output
JSON
items- The response items from the Bright Data API after requesting snapshot delivery.
Dependencies
- Bright Data API
Troubleshooting
- Ensure the Snapshot ID is provided; missing it will cause an error.
- Verify that all required credentials for the chosen delivery type are correctly set to avoid authentication failures.
- Check that the filename template and extension are provided and valid to prevent delivery errors.
- For JSON input fields like URLs or filters, ensure the JSON format is correct to avoid parsing errors.
- If delivery fails, inspect error messages from the Bright Data API for details on misconfiguration or network issues.
Links
- Bright Data Snapshots Documentation - Official documentation on managing and delivering snapshots with Bright Data.
- Bright Data Delivery Methods - Details on supported delivery types and configuration options for snapshot delivery.