Package Information
Documentation
n8n-nodes-sixdee-bm25
Full-text search inside n8n. No database, no embedding model, no external service.
Point it at your documents and search them with ranked, explainable results. The whole index
is one JSON file on disk.
| node | what it does |
|---|---|
| BM25 Index | Extracts text from a document, splits it into passages, writes them to a named index |
| BM25 Search | Ranks those passages against a query — with score, snippet and the terms that matched |
Install
In n8n: Settings → Community nodes → Install → n8n-nodes-sixdee-bm25
Self-hosted, manually:
cd ~/.n8n/nodes
npm install n8n-nodes-sixdee-bm25
Restart n8n. Two nodes appear under Data & Storage, plus BM25 Search Tool under
AI → Tools. Requires Node 20.15+ and a recent n8n (community nodes API v1).
Quick start
Read/Write Files From Disk ──▶ BM25 Index (index name: handbook)
↓ writes ~/.n8n/bm25/handbook.bm25.json
Chat Trigger / any node ──▶ BM25 Search (index name: handbook)
- Index. Put any node that produces a file in front of BM25 Index — Read/Write
Files From Disk, HTTP Request, Google Drive, S3, FTP, a Gmail attachment. Set an
Index Name. That is the whole configuration. - Search. Add BM25 Search with the same index name. The query comes from the
incoming item —query,chatInput,text,inputorq, whichever is present — so a
Chat Trigger wired straight into it works with nothing to fill in.
A hit looks like this:
{
"id": "handbook.pdf#4",
"score": 2.3125,
"title": "handbook.pdf — page 3",
"snippet": "…Prepaid customers may claim a refund within 30 days of a failed recharge…",
"text": "Prepaid customers may claim a refund within 30 days of a failed recharge. Refunds are credited…",
"matchedTerms": ["refund", "prepaid", "recharge"],
"metadata": { "documentId": "handbook.pdf", "page": 3, "chunk": 4, "team": "care" }
}
Why BM25 and not a vector store
BM25 matches on shared words, not meaning. That makes it the wrong tool for "find me
something about customer unhappiness" and the right one for product codes, error strings,
ticket numbers, policy names, people, and anything a user types the way the document spells
it. It needs no embedding model, so indexing is instant, free and private, and every hit is
explainable — the node tells you which terms matched.
Alongside a vector store it earns its place: BM25 catches the exact identifiers that
embeddings blur together.
Indexing
Input is one of:
- Binary File — the file on the item.
PDF,DOCX,HTML,XML,JSON,NDJSON,CSV,TSV,Markdownand plain text are read directly. Set the binary property to*
to index every file on the item. For other formats convert upstream (n8n's Extract from
File handles XLSX, ODS, RTF) and use the Text input. - Document Loader — whatever a loader sub-node connected to the Document port hands
over, already split by its own text splitter. - Text — an expression such as
{{ $json.content }}. - Item Fields — index the item JSON, naming which field is the title and which carry the
body.
To index a folder, put Read/Write Files From Disk in front with a glob
(/data/handbooks/**/*.pdf): it emits one item per file and the node indexes each.
Chunking
A 200-page PDF indexed whole matches every query and tells you nothing about where. The node
splits documents into passages, and search returns the passage.
| strategy | cuts on | good for |
|---|---|---|
paragraph (default) |
blank lines, packed to the chunk size | prose, handbooks, wikis |
characters |
fixed windows, preferring sentence ends | dense text with no paragraph structure |
heading |
Markdown headings, all-caps title lines | structured docs, runbooks |
page |
one chunk per page | PDFs where "page 14" is a useful answer |
line |
packed lines | logs, record-per-line exports |
none |
nothing | short documents, FAQ entries |
Chunk Size (1200 characters) and Chunk Overlap (120) sit on the node. Overlap applies
to every packing strategy, not just characters: each chunk after the first opens with the
tail of the one before it, so a passage straddling a boundary stays whole in one of them. It
is capped at half the chunk size and counts against the size budget.
Smaller chunks give more precise hits and more of them; larger ones keep more context.
600–1500 characters suits prose; logs do better with line and little or no overlap.
Document identity and metadata
Every chunk of a document shares one documentId. Re-indexing that ID replaces the previous
chunks instead of duplicating them, and deleting by it removes the whole document. Default:
the file name.
Whatever JSON you put in Metadata is stored on each chunk, returned with every hit, and
is what Filter matches on — put there what you will want to narrow by: source, tenant,
team, author, date.
The Document port
Like the vector store nodes, BM25 Index has a sub-node port. Connect n8n's Default Data
Loader (with a text splitter under it) and set Input to Document Loader:
Read Files From Disk ──▶ BM25 Index
△
│ Document
Default Data Loader
△
│ Text Splitter
Recursive Character Text Splitter
The loader has already read and split the document, so its chunks are indexed as they arrive
and its metadata (page, line range, source) rides along onto every hit.
Searching
| operation | returns |
|---|---|
| Search | ranked passages with score, snippet and matched terms |
| Get Document | one chunk by ID |
| Find Similar | more-like-this, from a document's most distinctive terms |
| List Documents | what is in the index, filterable |
| Index Info | document count, vocabulary, field weights |
| List Indexes | every index in the data directory |
mode says how a result was found: bm25 for a lexical match, fuzzy when the query
matched nothing and the trigram fallback caught a typo, filter for a query-less listing.
Options worth knowing: Filter (metadata equality, an array means "any of"), Search
Fields, Minimum Score, Snippet Length, Highlight Matches, and Combine Into
Single Item — one item holding every hit, which is the shape to hand an LLM.
With an AI Agent
The search node is flagged usableAsTool, so n8n publishes BM25 Search Tool and an agent
can attach it. One caveat worth understanding: n8n builds a tool's input schema purely from$fromAI() placeholders in a node's parameters, and this node takes its query from the item
rather than from a parameter. Two ways to give an agent real search:
- Retrieve, then prompt (recommended, deterministic):
Chat Trigger → BM25 Search → AI Agent, passing the passages in as context. No tool call, no chance of the model inventing
search terms. - As a tool: give it a parameter to fill, for example
={{ $fromAI('query', 'what to look for', 'string') }}in Filter, so the model has an
argument to pass.
Rename the node to something the model can reason about — search_support_handbook beatsBM25 Search: the node name becomes the tool name.
Where the index lives
One file per index: <data dir>/<name>.bm25.json, resolved in this order:
- the Data Directory option on the node,
N8N_BM25_DATA_DIR,~/.n8n/bm25— or$N8N_USER_FOLDER/.n8n/bm25when that is set, since n8n treats it as
the parent of.n8n.
Both nodes must point at the same directory. Inside .n8n it survives container restarts
with the rest of n8n's data.
Writes are atomic (temp file then rename) and serialised by an advisory lock, so two
executions — or two queue-mode workers — cannot interleave and lose documents. Reads are
cached per n8n process and revalidated against the file's mtime.
To check what an instance is really using, run Index Info or List Indexes: both
return the resolved dataDir. ({{ $env.N8N_BM25_DATA_DIR }} will not tell you — n8n blocks
environment access in expressions by default.)
Scoring
BM25F over two fields, title (weight 3) and text (weight 1), each with its own length
normalisation, k1 = 1.2. Change the weights with Field Weights when the index is first
created.
- Compound identifiers split on underscores, camelCase and digit/letter boundaries, so
CUST_360_DATA_30Dis reachable bydata,360or30d, andgetUserByIdbyuser. - Plurals are folded (
recharges→recharge). - An exact title match is boosted; a verbatim phrase, more gently.
- A query matching nothing falls back to character-trigram containment, so
refnudstill
finds the refund policy.
Scale
The index is held in memory and written whole — comfortable to roughly 100k chunks (a few
hundred MB of source documents). Beyond that, split by source or tenant, or turn on
Compress Index File. Searches are O(matching postings): single-digit milliseconds.
Development
npm install
npm run build # tsc + icons and codex files
npm test # 79 tests: engine, chunking, loaders, storage, both nodes
npm run lint # tsc --noEmit
Licence
MIT © Subin K Soman