Ido ToolboxInstall

Overview

The "Split Text to Chunks" operation in the "Other Tool" resource splits a large input text into smaller chunks of specified size, optionally allowing an overlap between consecutive chunks. This is useful when processing or analyzing large texts that need to be handled in parts, such as feeding text into APIs with size limits, preparing data for batch processing, or segmenting content for display.

Practical examples:

• Splitting a long article into paragraphs or sections for separate analysis.
• Breaking down a large document to send in multiple API requests where each request has a character limit.
• Creating overlapping text segments for context-aware processing in natural language tasks.

Properties

Output

The output JSON contains the following fields:

• inputText: The original input text provided.
• chunks: An array of strings, each representing a chunk of the original text split according to the chunk size and overlap size.
• totalChunks: The total number of chunks created.
• chunkSize: The chunk size parameter used.
• overlapSize: The overlap size parameter used.
• totalCharacters: The total number of characters in the original input text.

This output allows downstream nodes to process each chunk individually or collectively.

Dependencies

No external services or API keys are required for this operation. It runs entirely within the node's execution environment.

Troubleshooting

• Empty or missing input text: If the input text is empty or not provided, the output will contain zero chunks. Ensure the "Input Text" property is correctly set.
• Chunk size too small or zero: Setting the chunk size to zero or a very small number may cause unexpected behavior or many tiny chunks. Use a reasonable chunk size.
• Overlap size larger than chunk size: If the overlap size is equal to or greater than the chunk size, the logic may produce repeated chunks or infinite loops. Keep overlap size smaller than chunk size.
• Large input text performance: Very large input texts may increase execution time or memory usage; consider splitting input beforehand if needed.

Links and References

• JavaScript String.prototype.slice()
• n8n Documentation - Creating Custom Nodes
• General best practices for text chunking in NLP and API usage contexts.