Query + Full Scrape
AI-generatedOverview
This node performs web scraping and search operations using DuckDuckGo without requiring an API key. It supports searching for URLs based on a query, extracting content from a single page, filtering specific content types from a page, and performing a full scrape that combines searching and scraping multiple pages. It is useful for gathering web data such as URLs, page titles, headings, paragraphs, and bold text for further processing or analysis.
Use Case Examples
- Search DuckDuckGo for URLs related to a keyword and retrieve the top 5 result links.
- Scrape a single webpage to extract its title, headings, and paragraphs.
- Filter specific content types like headings or paragraphs from a given webpage URL.
- Perform a full scrape by searching DuckDuckGo for a query, then scraping the resulting URLs for detailed content extraction.
Properties
| Name | Meaning |
|---|---|
| Search Query | Text to search on DuckDuckGo for operations that involve searching (searchUrls, fullScrape). |
| Maximum URLs | The maximum number of result links to process from the search results (minimum 1, maximum 20). |
Output
JSON
url- The URL of the webpage or search result link.title- The title of the webpage, extracted from meta tags or the HTML title tag.h1- An array of H1 headings extracted from the webpage.paragraphs- An array of paragraph texts extracted from the webpage, filtered to exclude short content.query- The search query used when performing a search operation.urls- An array of URLs returned from the DuckDuckGo search results.error- Error message if an operation fails or no results are found.type- Type of error or special condition such as 'NoResults'.filterType- The type of content filter applied when filtering page content.data- Filtered content data extracted from the webpage based on the filter type (headings, paragraphs, bold text, or all).
Dependencies
- Uses the 'got' library for HTTP requests to fetch search results and webpage content.
Troubleshooting
- Common issues include no search results found for the query, resulting in an error with type 'NoResults'.
- Timeout errors may occur if a webpage takes too long to respond (timeout set to 5 seconds).
- Errors in scraping or parsing HTML content may result in error messages with details logged.
- Some URLs are filtered out if they belong to known social media or video platforms to avoid irrelevant results.