BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions14

Marketplace Dataset → Deliver Snapshot

AI-generated

Overview

This node interacts with the Bright Data platform to deliver snapshots of marketplace datasets to various delivery endpoints. It supports multiple delivery types including cloud storage services (Amazon S3, Aliyun OSS, Google Cloud Storage, Microsoft Azure), messaging services (Google Cloud PubSub), SFTP servers, Snowflake data warehouse, and webhooks. Users provide the snapshot ID and configure delivery parameters such as destination credentials, file naming templates, and file formats. The node sends a delivery request to Bright Data's API to initiate the snapshot delivery process.

Use Case Examples

  1. Deliver a marketplace dataset snapshot to an Amazon S3 bucket with specified AWS credentials and file naming conventions.
  2. Send a snapshot to a Google Cloud PubSub topic with client credentials and message attributes.
  3. Deliver a snapshot file to an SFTP server using SSH key authentication and custom remote path.
  4. Trigger delivery of a snapshot to a Snowflake stage with database and role credentials.

Properties

Name Meaning
snapshot_id The ID of the snapshot to operate on, required to specify which snapshot to deliver.
deliver_type The type of delivery endpoint to send the snapshot to, such as S3, webhook, Azure, etc.
endpoint Webhook URL to deliver the snapshot, required if deliver_type is webhook.
filename_template Template for the filename including placeholders, required for all delivery types.
filename_extension File extension for the delivered file, such as JSON, JSONL, or CSV.
topic_id Google PubSub topic ID, required if deliver_type is gcs_pubsub.
client_email Client email for Google Cloud credentials, required for gcs and gcs_pubsub delivery types.
private_key Private key for Google Cloud credentials, required for gcs and gcs_pubsub delivery types.
attributes Attributes to include in the PubSub message, optional for gcs_pubsub delivery.
container Name of the Azure container, required for Azure delivery.
bucket Name of the bucket for S3, Aliyun OSS, or Google Cloud Storage delivery.
aws-access-key AWS Access Key ID, required for S3 delivery.
aws-secret-key AWS Secret Access Key, required for S3 delivery.
access-key Access key for Aliyun OSS delivery.
secret-key Secret key for Aliyun OSS delivery.
account Azure storage account, required for Azure delivery.
key Azure storage key, required for Azure delivery.
sas_token Azure SAS token for access, required for Azure delivery.
role_arn Role ARN for AWS S3 delivery, optional.
external_id External ID for AWS S3 delivery, optional.
directory Target path or directory for delivery, optional for many delivery types.
region AWS or Aliyun region, optional for S3 and Aliyun OSS delivery.
host SFTP server host, required for SFTP delivery.
port SFTP server port, default is 22, optional for SFTP delivery.
path Remote path on the SFTP server to store the file, optional for SFTP delivery.
username SFTP username, required for SFTP delivery.
password SFTP password, required for SFTP delivery.
ssh_key SSH key for SFTP authentication, required for SFTP delivery.
passphrase Passphrase for the SSH key, optional for SFTP delivery.
database Snowflake database name, required for Snowflake delivery.
schema Snowflake schema name, required for Snowflake delivery.
stage Snowflake stage name, required for Snowflake delivery.
role Snowflake role, required for Snowflake delivery.
warehouse Snowflake warehouse, required for Snowflake delivery.
credentials.account Snowflake account, required for Snowflake delivery.
credentials.user Snowflake user, required for Snowflake delivery.
credentials.password Snowflake password, required for Snowflake delivery.
compress Whether to compress the response in gzip format, optional.

Output

JSON

  • items - The response items from the Bright Data API after requesting snapshot delivery.

Dependencies

  • Bright Data API

Troubleshooting

  • Common errors include missing required parameters such as snapshot ID or delivery credentials. Ensure all required fields for the selected delivery type are provided.
  • Invalid JSON format errors may occur when providing JSON input parameters like attributes or filters; validate JSON syntax before input.
  • API request failures may result from incorrect credentials or network issues; verify credentials and network connectivity.

Links

Discussion