Databricks Vector Store icon

Databricks Vector Store

Operations for Databricks Vector Store

Databricks Vector Store

AI-generated

Overview

This node provides operations for interacting with a Databricks Vector Store, enabling users to insert documents, retrieve ranked documents based on similarity queries, and use the retrieved documents as vector stores or tools in AI workflows. It is useful for managing and querying vector-based document stores in Databricks, supporting tasks like semantic search, document embedding, and AI tool integration.

Use Case Examples

  1. Insert a batch of documents into a Databricks vector store for later semantic search.
  2. Retrieve top-ranked documents from the vector store based on a similarity search prompt.
  3. Use retrieved documents as a vector store or tool in AI chains or agents for enhanced AI capabilities.

Properties

Name Meaning
Operation Mode Selects the mode of operation for the node, such as retrieving documents, inserting documents, or loading ranked documents.
Name Name of the vector store (required when using retrieve-as-tool mode).
Description Description of the tool to explain its purpose to the LLM (required when using retrieve-as-tool mode).
Index Name Name of the vector search index in Databricks (required).
Text Column Name of the column containing the document text (required).
Metadata Columns Comma-separated list of columns to include as metadata.
Score Threshold Minimum similarity score threshold for document retrieval.
Embedding Batch Size Number of documents to embed in a single batch (used in insert mode).
Prompt Search prompt to retrieve matching documents from the vector store using similarity-based ranking (used in load mode).
Limit Number of top results to fetch from vector store (used in load and retrieve-as-tool modes).
Include Metadata Whether or not to include document metadata in the output (used in load and retrieve-as-tool modes).
ID ID of an embedding entry (used in update mode).

Output

JSON

  • documents - Array of retrieved documents with their text and optional metadata.
  • toolName - Name of the vector store tool (when used as a tool).
  • toolDescription - Description of the tool for AI agent usage.

Dependencies

  • Databricks API credentials (host URL and token)

Troubleshooting

  • Error due to missing required input properties like indexName or textColumn - ensure these are provided.
  • Authentication errors if Databricks credentials are incorrect or missing - verify API token and host URL.
  • Score threshold too high may result in no documents returned - try lowering the threshold.
  • Batch size too large during insert may cause performance issues - reduce embedding batch size.

Links

Discussion