free-web-scrapping

Node n8n for searching and extracting content from web pages via DuckDuckGo and free scraping.

Package Information

Released: 15 mo ago
Downloads: 125 weekly / 328 monthly
Latest Version: 1.0.14
Author: Threndy

Available Nodes

Web Scraper (Free) icon
Web Scraper (Free)
Perform free DuckDuckGo searches or scrape pages for headings, paragraphs, and more (no API key)

Documentation

Web Scraper (Free)

A no-API-key, DuckDuckGo-based search & scraping node for n8n.
Perform searches and extract page content—all without paying for an API or maintaining credentials.

🚀 Features

  1. Search URLs only
    Perform a DuckDuckGo search and return the top N result URLs.

  2. Extract single page
    Fetch any web page and return its title, H1 headers, and paragraphs.

  3. Filter page content
    Fetch a page and return only the parts you care about:

    • Headings (H1, H2, H3)
    • Paragraphs
    • Bold text
    • All of the above

  4. Query + Full Scrape
    Combines “Search URLs only” and “Extract single page”:

    • Search DuckDuckGo for a query
    • Fetch and parse each resulting URL

  5. Browser-like HTTP requests

    • Custom User-Agent header
    • Configurable timeout to prevent hanging workflows

  6. Built-in blacklist
    Automatically skips links from social, video, and other noise domains (e.g. YouTube, Facebook, Instagram, TikTok, Reddit, etc.).

  7. Zero dependencies
    No external APIs or credentials required—just install once and start scraping.

📦 Installation

  1. In your n8n instance, open Settings → Community Nodes → Install
  2. In npm Package Name enter:

n8n-nodes-free-web-scrapping
  1. Click Install and restart n8n if prompted.

🔧 Configuration

When you add the Web Scraper (Free) node, you’ll see:

1. Operation

Value Description
searchUrls DuckDuckGo search → return URLs only
extractPage Fetch a single page → extract headers & paragraphs
filterContent Fetch a single page → return only filtered content
fullScrape DuckDuckGo search → fetch & extract content of URLs

2. Parameters by Operation

Search URLs only & Full Scrape

  • Search Query (string)
    Text to search for on DuckDuckGo.
  • Maximum URLs (number)
    How many top links to process (1–20).

Extract single page & Filter page content

  • Page URL (string)
    The URL of the page you want to scrape.

Filter page content only

  • Filter Type (options)
  • headings → H1, H2, H3
  • paragraphs → <p> elements
  • bolds → <b> and <strong>
  • all → combinations of the above

📖 Examples

1. Search URLs only

[
{ "url": "https://openai.com/gpt-4" },
{ "url": "https://en.wikipedia.org/wiki/GPT-4" }
]

2. Extract single page

[
  {
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "title": "Web scraping - Wikipedia",
    "h1": ["Web scraping"],
    "paragraphs": [
      "Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites..."
    ]
  }
]

3. Filter page content

[
  {
    "url": "https://example.com",
    "filterType": "headings",
    "data": {
      "h1": ["Main Heading"],
      "h2": ["Section Title", "Another Section"],
      "h3": ["Subsection"]
    }
  }
]

4. Query + Full Scrape

[
  {
    "url": "https://example.com/page1",
    "title": "Example Page",
    "h1": ["Example Heading"],
    "paragraphs": ["First paragraph...", "Second paragraph..."]
  }
]

🔍 Blacklist

youtube.com, instagram.com, facebook.com, twitter.com,
tiktok.com, snapchat.com, linkedin.com, pinterest.com,
reddit.com, twitch.tv, vimeo.com, dailymotion.com,
bilibili.com, youku.com

⚙️ Under the Hood

  • Uses got for HTTP requests.
  • Parses DuckDuckGo’s HTML endpoint:
    https://html.duckduckgo.com/html?q=<query>
  • Extracts links via regex on the result__a class.
  • Cleans HTML with regex-based strip functions (scripts, styles, comments, tags).
  • All operations return JSON—ready for downstream nodes.

Discussion