Database Importer icon

Database Importer

Import data from Kingsoft node output to MySQL database with field mapping and deduplication

Database Importer

AI-generated

Overview

This node imports data from Kingsoft node output into a MySQL database table with support for field mapping, batch processing, and deduplication. It supports two import modes: full import, which clears the target table before importing all data, and incremental import, which imports only new records based on a unique key combination. The node is useful for synchronizing data from Kingsoft outputs into a MySQL database efficiently, handling large datasets in batches, and ensuring data integrity by avoiding duplicates.

Use Case Examples

  1. Full import of all data from Kingsoft output into a specified MySQL table, truncating the table before import.
  2. Incremental import where only new records are added based on unique key fields, avoiding duplicates.
  3. Mapping source fields from Kingsoft output to different target database fields during import.
  4. Batch processing large datasets with configurable batch size for performance optimization.

Properties

Name Meaning
Import Mode Selects the import mode: full import clears the target table and imports all data; incremental import imports only new records based on unique keys.
Target Table Name of the target MySQL database table to import data into, supports database.table format.
Batch Size Number of records to import in each batch, configurable between 100 and 10,000.
Field Mappings Mappings from source fields in Kingsoft output to target database fields, allowing flexible data transformation during import.
Unique Key Fields Comma-separated list of target field names that form the unique key for incremental import to identify new records.
Options Additional options including whether to truncate the table before full import, whether to continue on error during batch processing, and whether to verify that the number of imported records matches the source data.

Output

JSON

  • success - Boolean indicating if the import was successful.
  • totalRows - Total number of rows processed from the source data.
  • importedRows - Number of rows successfully imported into the database.
  • duplicateRows - Number of duplicate rows detected and skipped during incremental import.
  • failedRows - Number of rows that failed to import.
  • message - Summary message describing the import result including counts and performance.

Dependencies

  • MySQL database connection using credentials with host, port, user, password, and optional database name.

Troubleshooting

  • Ensure that at least one field mapping is provided; otherwise, the node throws an error.
  • Source data must be an array with at least a header row and one data row; otherwise, an error is thrown.
  • If the target table does not exist or credentials are incorrect, the node will fail to connect or execute queries.
  • Batch insert failures may occur due to data type mismatches or constraint violations; enabling 'Continue on Error' allows processing to continue despite batch failures.
  • Verify that unique key fields are correctly specified for incremental imports to avoid incorrect duplicate detection.

Discussion