Vector Store Processor icon

Vector Store Processor

Preprocess documents from any source (PDF, LLMs, etc.) for vector stores with intelligent chunking

Vector Store Processor

AI-generated

Overview

The Vector Store Processor node preprocesses documents from various sources (e.g., PDFs, LLM outputs) by intelligently chunking text into manageable pieces for vector stores. It supports combining all input items into one document or processing each item separately, allowing flexible document handling. This node is useful for preparing large texts for vector-based search or machine learning applications, ensuring chunks are within specified size limits and enriched with metadata.

Use Case Examples

  1. Combining multiple text inputs into a single document, chunking it, and adding metadata for vector search indexing.
  2. Processing each input document separately to create chunks with custom metadata fields for individual vector store entries.

Properties

Name Meaning
Chunk Size Maximum characters per chunk to split the document into.
Min Chunk Size Minimum characters per chunk to ensure chunks are not too small.
Processing Mode Determines whether to combine all input items into one document for processing or process each item separately.
Max Chunks Per Execution Maximum number of chunks to output per execution; 0 means unlimited. Recommended 10-15 for stability with vector stores.
Namespace Max Length Maximum length allowed for the namespace string used in metadata.
Flush Memory After Processing Whether to force garbage collection after processing to free up memory.
Additional Metadata Fields Custom metadata fields to add to each chunk, specified as name-value pairs.

Output

JSON

  • pageContent - Text content of the chunk.
  • metadata
    • document_title - Title of the document or chunk.
    • chapter - Chapter name, default is 'Main Content'.
    • section - Section name, default is 'Document Section'.
    • content_type - Type of content, e.g., 'general' or 'empty'.
    • chunk_index - Index of the chunk in the entire document.
    • local_chunk_index - Local index of the chunk within the current processing context.
    • chapter_index - Index of the chapter, default is 0.
    • total_chunks - Total number of chunks created from the document.
    • namespace - Namespace string derived from the document title or metadata.
    • source_file - Source file name or identifier.
    • character_count - Number of characters in the chunk.
    • processing_timestamp - Timestamp when the chunk was processed.
    • item_index - Index of the input item when processing each item separately.

Troubleshooting

  • If the node outputs empty chunks, ensure the input items contain valid text fields such as 'text', 'markdown', or 'content'.
  • If garbage collection is enabled but not working, run Node.js with the '--expose-gc' flag to enable memory flushing.
  • Large documents may produce many chunks; adjust 'Max Chunks Per Execution' to balance performance and stability.

Discussion