TorqData DatasetInstall

Overview

This node splits input text into smaller chunks, generates vector embeddings for each chunk using OpenAI embedding models, and indexes these embeddings into an OpenSearch index managed by TorqData. It is designed to facilitate semantic search or similarity search over large text datasets by converting text into high-dimensional vectors that can be efficiently queried.

Common scenarios where this node is beneficial include:

• Preparing textual data for semantic search applications.
• Indexing documents or knowledge bases to enable fast similarity queries.
• Enhancing chatbot or AI assistant capabilities by embedding context data.
• Integrating with TorqData's vector store infrastructure for scalable vector search.

Practical example: You have a large document or set of documents you want to make searchable by meaning rather than keywords. You input the text, select an embedding model, specify chunk size and overlap, and the node will split the text, embed each chunk, and index them into an OpenSearch vector index. Later, you can query this index to find semantically similar text segments.

Properties

Output

The node outputs an array of JSON objects, one per input item, each containing:

• text: The original input text.
• segments: An array of text chunks obtained by splitting the input text.
• embeddings: An array of objects, each with a vector field containing the embedding vector for the corresponding text segment.

No binary data output is produced.

Example output structure for one item:

{
  "text": "Full input text here...",
  "segments": [
    "First chunk of text",
    "Second chunk of text",
    "... more chunks ..."
  ],
  "embeddings": [
    { "vector": [0.123, -0.456, ...] },
    { "vector": [0.789, -0.012, ...] },
    "... more vectors ..."
  ]
}

Dependencies

• Requires an API key credential for TorqData API to fetch datasets and manage vector stores.
• Requires an OpenAI API key credential to generate embeddings using OpenAI models.
• Requires access to an OpenSearch instance (host URL configurable) to create indices and index vector data.
• Environment variable OPENSEARCH_HOST can be used to set default OpenSearch host URL.

Troubleshooting

• Missing API keys: The node throws errors if either the TorqData API key or OpenAI API key is missing. Ensure both credentials are configured properly.
• Dataset not found: If the selected dataset ID does not exist or is inactive, an error is raised. Verify dataset availability in TorqData.
• OpenSearch host missing or unreachable: If no OpenSearch host is provided or the host is unreachable, indexing fails. Confirm the host URL and network connectivity.
• Failed to create vector store: Errors during vector store creation in TorqData may occur due to API issues or invalid parameters. Check API responses and permissions.
• Embedding generation failure: If embedding generation fails for any chunk, the node reports which item failed. Possible causes include invalid model names or OpenAI API issues.
• JSON Data parsing errors: If additional JSON data is malformed, the node attempts fallback parsing but may log warnings. Validate JSON input format.
• Index mapping issues: The node attempts to create an OpenSearch index with KNN vector mappings if it doesn't exist. Failure here may indicate insufficient OpenSearch privileges or incompatible version.

Links and References

• TorqData API Documentation
• OpenAI Embeddings Models
• OpenSearch Vector Search
• n8n Expressions Documentation