n8n-nodes-flycrawl

Official FlyCrawl Community Node for n8n

High-Speed Web Scraping, Clean Fit-Markdown, and Native 1536/3072-dim Vector Embeddings for AI Agents & RAG Pipelines.

⚡ Key Capabilities

• Sub-350ms p50 Latency: Ultra-fast cloud scraping engine powered by distributed browser clusters.
• 83.2% Token Slashed (Fit-Markdown): Eliminates scripts, tracking bloat, CSS, and navigation clutter before sending content to LLMs.
• Native 1536 & 3072-dim Embeddings: Scrape and convert text into normalized vector embeddings directly inside n8n without external OpenAI embedding calls.
• Autonomous AI Agent Tool: Flagged with usableAsTool: true for direct pairing with n8n AI Agent nodes (OpenAI, Claude, LangChain).
• Stealth Anti-Bot & Residential Proxies: Built-in Cloudflare Turnstile bypass and 10+ residential country proxy locations (US, DE, GB, FR, JP, AE, etc.).
• Zero Host Overhead: Headless Chrome runs on FlyCrawl's distributed infrastructure, preserving your n8n server's RAM and CPU.

📦 Installation

Option 1: Via n8n Web Interface (Recommended)

1. Open your n8n instance and navigate to Settings > Community Nodes.
2. Click Install a community node.
3. Enter n8n-nodes-flycrawl into the npm package name field.
4. Agree to the risks and click Install.
5. The FlyCrawl node is now available on your workflow canvas!

Option 2: Docker / Self-Hosted n8n

Inside your custom n8n Docker image or volume root:

cd /home/node/.n8n
npm install n8n-nodes-flycrawl

Restart your n8n container, and the node will be automatically registered.

🔑 Credentials Configuration

1. In n8n, go to Credentials > New Credential.
2. Search for FlyCrawl API.
3. Enter your API Key (e.g. fc-xxxxxxxxxxxxxxxxxxxxxxxx).
   • Sign up at flycrawl.net to obtain your API key with 100 free credits.
4. Set Base URL (default is https://flycrawl.net/api/v1).
5. Click Save & Test. The credential will verify connection against the FlyCrawl endpoint.

🚀 Supported Operations

1. Scrape Single URL (scrape)

Converts any web page (including client-side JavaScript Single Page Apps) into clean, LLM-ready Fit-Markdown.

• Parameters:
  • url (string, required): The target webpage URL.
  • additionalFields.bypassAntiBot (boolean): Solve Cloudflare & CAPTCHAs automatically.
  • additionalFields.country (options): Select residential proxy location.
  • additionalFields.formats (multi-options): Fit-Markdown, Clean HTML, Extracted Tables.

2. Scrape & Vectorize (vectorize)

Extracts content, performs semantic markdown chunking (split across H1-H3 headers), and outputs vector embeddings directly compatible with Qdrant, Pinecone, Chroma, and pgvector.

• Parameters:
  • embeddingModel:
    • FlyCrawl Native Fast (1536-dim)
    • FlyCrawl Native Large (3072-dim)
  • chunkSize (number, default: 500): Target token count per chunk.

3. Deep Crawl Website (crawl)

Recursively traverses internal hyperlinks across an entire domain up to the specified depth.

• Parameters:
  • depth (number, default: 1): 0 = single page, 1 = direct links, 2+ = recursive links.
  • maxPages (number, default: 10): Maximum number of pages to crawl.

🤖 Using FlyCrawl as an AI Agent Tool

The FlyCrawl node implements usableAsTool: true. You can attach it directly to an AI Agent or OpenAI Assistant node in n8n.

When your autonomous agent needs real-time web access, documentation lookup, or deep search, it invokes FlyCrawl automatically and receives compressed Fit-Markdown without polluting its context window.

🛠️ Development & Building Locally

To build from source or contribute:

# Clone the repository
git clone https://github.com/FlyCrawl-ai/n8n-nodes-flycrawl.git
cd n8n-nodes-flycrawl

# Install dependencies
npm install

# Compile TypeScript and copy SVG icons to dist/
npm run build

The compiled output will be generated inside the dist/ directory:

• dist/credentials/FlyCrawlApi.credentials.js
• dist/nodes/FlyCrawl/FlyCrawl.node.js
• dist/nodes/FlyCrawl/flycrawl.svg

📄 License

MIT © FlyCrawl Engineering