Overview

This node performs web scraping operations, specifically filtering content from a given web page URL. It fetches the HTML content of the specified page and extracts parts of the content based on the selected filter type, such as headings (h1, h2, h3), paragraphs, bold text, or all content. This is useful for scenarios where you want to programmatically extract specific types of information from web pages without needing an API key, for example, gathering article headings, summaries, or emphasized text from news sites or blogs.

Use Case Examples

1. Extract all headings from a news article to create a summary index.
2. Retrieve all paragraphs from a blog post for content analysis.
3. Get all bold text from a product page to highlight key features.

Properties

Output

JSON

• url - The URL of the scraped page.
• filterType - The type of content filter applied (headings, paragraphs, bolds, or all).
• data
  • h1 - Extracted h1 headings (if filter includes headings or all).
  • h2 - Extracted h2 headings (if filter includes headings or all).
  • h3 - Extracted h3 headings (if filter includes headings or all).
  • paragraphs - Extracted paragraphs (if filter includes paragraphs or all).
  • bolds - Extracted bold or strong text (if filter includes bolds or all).
• error - Error message if scraping or extraction fails.
• type - Type of error if an error occurs.

Dependencies

• Uses the 'got' library for HTTP requests to fetch web page content.

Troubleshooting

• Common issues include network timeouts or unreachable URLs, which result in error messages indicating request failures.
• If the page structure changes or contains heavy JavaScript rendering, the node might not extract content correctly since it relies on static HTML parsing.
• Errors related to invalid URLs or unsupported content types may occur; ensure the URL is correct and accessible.
• If no content matches the filter type, the output data fields may be empty or missing.

Links

• Got HTTP Request Library - The node uses 'got' for making HTTP requests to fetch web page content.