Web Scraper (Free) icon

Web Scraper (Free)

Perform free DuckDuckGo searches or scrape pages for headings, paragraphs, and more (no API key)

Actions4

Search URLs only

AI-generated

Overview

This node performs web scraping tasks focused on DuckDuckGo search results and webpage content extraction. Specifically, for the 'Search URLs only' operation, it takes a search query, performs a DuckDuckGo search, and returns a list of URLs from the search results, excluding popular social media and video sites. This is useful for workflows that need to gather relevant URLs based on a search term without requiring an API key. For example, it can be used to collect URLs for further processing or analysis in automation workflows.

Use Case Examples

  1. Searching for URLs related to a product or topic to gather sources for content aggregation.
  2. Automating the collection of URLs for monitoring news or blog posts on a specific subject.

Properties

Name Meaning
Search Query The text string to search on DuckDuckGo to find relevant URLs.
Maximum URLs The maximum number of URLs to return from the search results, limited between 1 and 20.

Output

JSON

  • url - The URL of a search result from DuckDuckGo.

Dependencies

  • Uses the 'got' HTTP client library to perform web requests.

Troubleshooting

  • If no results are returned, the output includes an error field with 'NoResults' indicating no URLs matched the query.
  • Network or request timeout errors may occur if the target website is unreachable or slow; these are logged and returned as error messages in the output.
  • URLs from popular social media and video platforms are filtered out and will not appear in results, which might cause fewer URLs than requested to be returned.

Links

  • DuckDuckGo HTML Search - The node uses DuckDuckGo's HTML search interface to retrieve search results without requiring an API key.

Discussion