BrightData icon

BrightData

Interact with Bright Data to scrape websites or use existing datasets from the marketplace to generate adapted snapshots

Actions16

Marketplace Dataset → Filter Dataset

AI-generated

Overview

The BrightData node's 'Filter Dataset' operation allows users to filter records from a selected marketplace dataset based on specified criteria. It supports both single and group filters, enabling users to apply simple or complex filtering logic to extract relevant data. This operation is useful for scenarios where users want to retrieve a subset of data from large datasets, such as filtering customer records by age or location, or extracting specific industry data from a marketplace dataset.

Use Case Examples

  1. Filtering a dataset to include only records where the 'industry' field equals 'Advertising'.
  2. Applying multiple filters to a dataset, such as records where 'age' is greater than 30 and 'country' equals 'USA'.
  3. Limiting the number of records returned from a dataset to 100 for performance optimization.

Properties

Name Meaning
Dataset Select the marketplace dataset to filter on. Supports selection from a list of available datasets.
Records Limit The maximum number of records to include in the filtered snapshot.
Filter Type Type of filter to apply: either a single filter or a group of filters combined with 'and' logic.
Field Name The name of the field to filter on (used when filter type is single filter).
Operator The operator to use for the filter condition (used when filter type is single filter).
Field Value The value to filter on (used when filter type is single filter).
Filters Group A JSON object representing a group of filters combined with 'and' logic (used when filter type is group filters).
CSV Filter A CSV formatted string filter (used when filter type is csv_filter).
JSON Filter A JSON formatted filter array (used when filter type is json_filter).
Request Options Additional request options such as batching, SSL certificate validation, proxy, and timeout settings.

Output

JSON

  • filteredRecords - The filtered dataset records based on the applied filter criteria.

Dependencies

  • Bright Data API key credential

Troubleshooting

  • Ensure the selected dataset ID is valid and accessible with the provided API credentials.
  • Verify the filter JSON syntax when using group filters to avoid parsing errors.
  • Check network connectivity and proxy settings if requests fail or time out.
  • Adjust the records limit to avoid performance issues with very large datasets.

Links

Discussion