Search URLs only
AI-generatedOverview
This node performs web scraping tasks focused on DuckDuckGo search results and webpage content extraction. Specifically, for the 'Search URLs only' operation, it takes a search query, performs a DuckDuckGo search, and returns a list of URLs from the search results, excluding popular social media and video sites. This is useful for workflows that need to gather relevant URLs based on a search term without requiring an API key. For example, it can be used to collect URLs for further processing or analysis in automation workflows.
Use Case Examples
- Searching for URLs related to a product or topic to gather sources for content aggregation.
- Automating the collection of URLs for monitoring news or blog posts on a specific subject.
Properties
| Name | Meaning |
|---|---|
| Search Query | The text string to search on DuckDuckGo to find relevant URLs. |
| Maximum URLs | The maximum number of URLs to return from the search results, limited between 1 and 20. |
Output
JSON
url- The URL of a search result from DuckDuckGo.
Dependencies
- Uses the 'got' HTTP client library to perform web requests.
Troubleshooting
- If no results are returned, the output includes an error field with 'NoResults' indicating no URLs matched the query.
- Network or request timeout errors may occur if the target website is unreachable or slow; these are logged and returned as error messages in the output.
- URLs from popular social media and video platforms are filtered out and will not appear in results, which might cause fewer URLs than requested to be returned.
Links
- DuckDuckGo HTML Search - The node uses DuckDuckGo's HTML search interface to retrieve search results without requiring an API key.