unified-scraper

Generic n8n node for any scraper service with custom URL and API key

Package Information

Released: 4 mo ago
Downloads: 29 weekly / 153 monthly
Latest Version: 4.1.0
Author: MyMP

Available Nodes

Unified Scraper
Generic unified scraper - reusable for any scraper service

Documentation

@mymp/n8n-nodes-unified-scraper v4.0.0

Unified n8n community node for 12 scrapers with automatic database insertion.

Supported Scrapers (v4.0.0)

  • Scraper X (Twitter/X)
  • Scraper Facebook
  • Scraper Instagram
  • Scraper LinkedIn
  • Scraper TikTok
  • Scraper Threads
  • Scraper News
  • Scraper News Go
  • Scraper Google News
  • Scraper Reddit Go
  • Scraper Google Search
  • Instagram API

New in v4.0.0

  • 12 Scrapers: Expanded from 3 to 12 scrapers
  • DB Insertion: Optional automatic insertion to scraper-db-service
  • Retry Logic: Configurable retries (0-5) with exponential backoff
  • PostgreSQL Credential: Direct DB access support
  • Standardized Operations: All scrapers support search, trending, getById

Usage

Basic Scraping

  1. Add the Unified Scraper node to your workflow
  2. Select your scraper and operation
  3. Enter your query/ID
  4. (Optional) Enable "Insert to Database"
  5. Execute

With Database Insertion

  1. Configure credentials:
    • Unified Scraper API (DB service API key required)
    • PostgreSQL (optional, for direct queries)
  2. Enable "Insert to Database" in node parameters
  3. Results will be automatically stored

Credentials

Unified Scraper API

  • Scraper DB Service Base URL (default: http://scraper-db-service:8088)
  • Scraper DB Service API Key (required for DB insert)
  • Individual scraper API keys (optional)

PostgreSQL

  • Host, Port, Database, User, Password

Output Format

{
  "success": true,
  "scraper": "scraper-x",
  "operation": "search",
  "results": [...],
  "dbInserted": true,
  "dbStats": {
    "added": 10,
    "updated": 2,
    "skipped": 0
  },
  "errors": [],
  "retryCount": 0
}

📦 Installation

Method 1: Direct Copy (Recommended)

# Clone the repository
git clone https://github.com/mymp/n8n-nodes-unified-scraper.git
cd n8n-nodes-unified-scraper

# Build the node
npm install
npm run build

# Copy to n8n custom nodes directory
cp -r dist/* /path/to/n8n/custom-nodes/

# Restart n8n
docker-compose restart
# or
npm restart

Method 2: From NPM (Coming Soon)

cd /path/to/n8n/custom-nodes
npm install @mymp/n8n-nodes-unified-scraper

For Docker Users

Add to your docker-compose.yml:

services:
  n8n:
    image: n8nio/n8n
    volumes:
      - ./custom-nodes:/custom-nodes
    environment:
      - N8N_CUSTOM_NODES_DIR=/custom-nodes
      - NODE_FUNCTION_ALLOW_BUILTIN=crypto,fs

🔑 Configuration

Step 1: Create Credentials

  1. In n8n UI, go to Credentials → New
  2. Select Unified Scraper API
  3. Configure all platforms:

Scraper X (Twitter/X):

  • Scraper X Base URL: http://localhost:8082
  • Scraper X API Key: (leave empty if no auth)

Scraper Facebook:

  • Scraper Facebook Base URL: http://localhost:8083
  • Scraper Facebook API Key: (leave empty if no auth)

Instagram API:

  • Instagram API Base URL: http://localhost:8080
  • Instagram API Key: (leave empty if no auth)

Scraper DB Service (v4.0.0):

  • Scraper DB Service Base URL: http://scraper-db-service:8088
  • Scraper DB Service API Key: (required for DB insert)

Step 2: Create Workflow

  1. Add a new node → Unified Scraper
  2. Select platform (12 options in v4.0.0)
  3. Select operation (options change based on platform)
  4. Configure parameters
  5. Enable "Insert to Database" (optional, v4.0.0)
  6. Connect to other nodes

📖 Usage Examples

Example 1: Get Malaysia Trending Topics

{
  "scraper": "scraper-x",
  "operation": "trending",
  "region": "malaysia",
  "limit": 20,
  "headless": true,
  "insertToDatabase": true,
  "retryCount": 3
}

Example 2: Scrape Facebook Page with DB Insert

{
  "scraper": "scraper-facebook",
  "operation": "page",
  "pageUrl": "https://www.facebook.com/meta",
  "limit": 20,
  "headless": true,
  "insertToDatabase": true
}

Example 3: Instagram Login + Get User Info

// Node 1: Login
{
  "scraper": "instagram-api",
  "operation": "igLogin",
  "igUsername": "your_username",
  "igPassword": "your_password"
}

// Node 2: Get User Info
{
  "scraper": "instagram-api",
  "operation": "igUser",
  "igUsername": "natgeo",
  "insertToDatabase": true
}

Example 4: LinkedIn Search (v4.0.0)

{
  "scraper": "scraper-linkedin",
  "operation": "search",
  "query": "software engineer",
  "limit": 25,
  "insertToDatabase": true,
  "retryCount": 2
}

Example 5: Multi-Platform Monitoring with DB

Cron Trigger (every 30 minutes)
  ↓
Split in Batches
  ↓     ↓          ↓           ↓
X Trending   FB Page   IG Hashtag   LinkedIn Jobs
  ↓           ↓          ↓             ↓
          Merge ←──────┴─────────────┘
            ↓
      IF (high engagement)
            ↓
    Insert to Database
            ↓
      Slack (alert)

🎛️ Node Parameters

Common Parameters (All Operations)

Parameter Type Default Description
scraper Options scraper-x Platform selector (12 options)
limit Number 20 Max items to return (1-100)
headless Boolean true Run browser headless (X/FB/LinkedIn only)
insertToDatabase Boolean false Insert results to DB (v4.0.0)
retryCount Number 0 Retry attempts with backoff (0-5, v4.0.0)

Scraper-X Parameters

Operation Parameters
trending region (malaysia, singapore, indonesia, united-states, india, uk, canada, australia, japan, philippines, thailand)
search keyword
comments tweetUrl
crawl entityName, keyword, sources
getById id (v4.0.0 standardized)
demoTrending region
demoSearch keyword

Scraper-Facebook Parameters

Operation Parameters
page pageUrl
group groupUrl
search query
post postUrl, includeComments
watch —
hashtag hashtag
getById id (v4.0.0 standardized)

Instagram API Parameters

Operation Parameters
igLogin igUsername, igPassword
ig2fa igUsername, ig2faCode
igAuthStatus —
igUser igUsername
igUserFeed igUsername, limit
igFollowers igUsername, limit
igFollowing igUsername, limit
igFollow / igUnfollow igUsername
igMedia igMediaId
igMediaByUrl igUrl
igLike / igUnlike igMediaId
igComments igMediaId, limit
igComment igMediaId, igCommentText
igSave igMediaId
igDmThreads limit
igDmMessages igThreadId, limit
igDmSend igUserId or igDmUsername, igDmText
igTimeline / igExplore limit
igSearchUsers / igSearchHashtags igQuery, limit
igHashtag igHashtag, igTab (top/recent), limit
igBatchLike igMediaIds (comma-separated), igDelay
igBatchFollow igUsernames (comma-separated), igDelay
igWebhookRegister igWebhookUrl, igWebhookEvents, igWebhookSecret
igWebhookList —
igWebhookTest igWebhookUrl, igWebhookSecret
igWebhookDelete igWebhookId

Additional Scrapers (v4.0.0)

Scraper Operations
Scraper LinkedIn search, trending, getById
Scraper TikTok search, trending, getById
Scraper Threads search, trending, getById
Scraper News search, trending, getById
Scraper News Go search, trending, getById
Scraper Google News search, trending, getById
Scraper Reddit Go search, trending, getById
Scraper Google Search search, trending, getById

🌐 API Endpoints

Scraper-X API

  • Base URL: http://localhost:8082
  • Auth: X-API-Key header or Authorization: Bearer <token>
  • Endpoints: POST /trending, POST /search, POST /comments, POST /crawl, POST /demo/trending, POST /demo/search

Scraper-Facebook API

  • Base URL: http://localhost:8083
  • Auth: X-API-Key header or Authorization: Bearer <token>
  • Endpoints: POST /page, POST /group, POST /search, POST /post, POST /watch, POST /hashtag

Instagram API

  • Base URL: http://localhost:8080
  • Auth: X-API-Key header or Authorization: Bearer <token>
  • Swagger UI: http://localhost:8080/docs
  • Endpoints: 41 routes across Auth, User, Media, Upload, Direct, Feed, Search, Hashtag, Batch, Webhooks

Scraper DB Service (v4.0.0)

  • Base URL: http://scraper-db-service:8088
  • Auth: X-API-Key header (required for writes)
  • Swagger UI: http://scraper-db-service:8088/docs
  • Endpoints: POST /scrape, GET /health, GET /stats

🧪 Testing

Test Scraper-X

curl -X POST http://localhost:8082/trending \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"region": "malaysia", "limit": 10}'

Test Scraper-Facebook

curl -X POST http://localhost:8083/page \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"page_url": "https://www.facebook.com/meta", "limit": 10}'

Test Instagram API

# Login
curl -X POST http://localhost:8080/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "your_user", "password": "your_pass"}'

# Get user info
curl http://localhost:8080/user/natgeo \
  -H "X-API-Key: your-key"

# Register webhook
curl -X POST http://localhost:8080/webhooks \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://your-server.com/webhook", "events": ["message"]}'

Test DB Service (v4.0.0)

# Health check
curl http://scraper-db-service:8088/health

# Scrape with DB insert
curl -X POST http://scraper-db-service:8088/scrape \
  -H "X-API-Key: your-db-key" \
  -H "Content-Type: application/json" \
  -d '{
    "scraper": "scraper-x",
    "operation": "trending",
    "params": {"region": "malaysia", "limit": 10},
    "insertToDb": true
  }'

🍪 Cookie Requirements

For authenticated scraping (Facebook groups, X search, Instagram):

Save Cookies via API (X/FB)

# Scraper-X
curl -X POST http://localhost:8082/auth/save \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"cookies": [...], "session_name": "x-session"}'

# Scraper-Facebook
curl -X POST http://localhost:8083/auth/save \
  -H "X-API-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"cookies": [...], "session_name": "fb-session"}'

Instagram Auth

Instagram API uses username/password login with session persistence. Just use the igLogin operation — sessions are saved automatically in ./sessions/.

🐛 Troubleshooting

Node Not Showing

  1. Restart n8n completely
  2. Check files are in /custom-nodes/ directory
  3. Check n8n logs for compilation errors
  4. Verify TypeScript build completed successfully (npm run build)

401 Unauthorized

  • Verify API keys are correct in credentials
  • Test API endpoints with curl first
  • Check base URLs are accessible
  • Verify authentication format (X-API-Key header)
  • For DB insert, verify Scraper DB Service API Key

Empty Results

  • Check if logged in (for authenticated endpoints)
  • Verify URLs/usernames are correct and public
  • Try with headless: false to debug (X/FB only)
  • Check API logs for errors
  • Verify retryCount is not causing delays

DB Insert Fails (v4.0.0)

  • Verify Scraper DB Service is running (curl http://scraper-db-service:8088/health)
  • Check API key is valid for writes
  • Verify data format matches expected schema
  • Check logs for constraint violations

📁 File Structure

n8n-nodes-unified-scraper/
├── nodes/
│   └── UnifiedScraperNode.ts       # Main unified node (12 scrapers)
├── credentials/
│   ├── UnifiedScraperApi.ts        # Credentials for all platforms
│   └── PostgreSQL.ts                # PostgreSQL credential (v4.0.0)
├── package.json                     # NPM package config
├── tsconfig.json                    # TypeScript config
├── README.md                        # This file
├── LICENSE                          # MIT License
└── .gitignore                      # Git ignore

🚀 Production Status

Component Status URL
Scraper-X API ✅ Ready http://localhost:8082
Scraper-Facebook API ✅ Ready http://localhost:8083
Instagram API ✅ Ready http://localhost:8080
Scraper DB Service ✅ Ready http://scraper-db-service:8088
n8n Node ✅ Ready v4.0.0

📄 License

MIT License — see LICENSE file for details

🤝 Contributing

Contributions welcome! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Test thoroughly with all platforms
  5. Submit a pull request

🎉 Summary

One node, twelve platforms, endless possibilities!

  • ✅ Single unified interface for 12 scrapers
  • ✅ 60+ operations across all platforms
  • ✅ Automatic database insertion with retry logic
  • ✅ Webhook support for Instagram events
  • ✅ Full TypeScript support
  • ✅ Easy installation

Discussion