Chonkie icon

Chonkie

Chunk text efficiently using Chonkie for RAG and NLP pipelines

Chonkie

AI-generated

Summary

Split input text into multiple chunks using one of three chunking strategies: Recursive, Token-based, or Fast, according to user-defined parameters.

Inputs

  • text (required) — The text string you want to divide into chunks.
  • chunkerType (required) — The chunking strategy to apply: Recursive (hierarchy-aware by paragraphs/sentences/words), Token (fixed token count with overlap), or Fast (WASM-based byte-level splitting).
  • chunkSize (required) — Maximum size of each chunk, interpreted as tokens or characters depending on the chunker.
  • chunkOverlap — (Token chunker only) Number of overlapping units between consecutive chunks.
  • delimiters — (Fast chunker only) Characters that define boundaries for splitting the text.
  • minCharactersPerChunk — (Recursive chunker only) Minimum size threshold to keep chunks from recursive splitting before merging.

Output shape

a list of JSON objects, each representing a chunk of the original text

Each output item contains 'text', 'tokenCount', 'startIndex', and 'endIndex' fields describing the chunk content and position in the input. Items are paired to their input index. If an error occurs per item and 'Continue On Fail' is enabled, an error message is returned instead of chunk data.

Examples

Example 1: Chunk input text using the Token chunker with a chunk size of 512 tokens and 50 token overlap.

Provide the full text, set chunkerType to 'token', chunkSize to 512, and chunkOverlap to 50. The node outputs multiple chunks with overlapping tokens as specified.

Links

Discussion