Process Raw HTML
AI-generatedOverview
This node, named "Crawl4AI: Basic Crawler," is designed to process raw HTML content by crawling and extracting meaningful data from it. It is particularly useful for scenarios where you have raw HTML input and want to parse, filter, and extract specific parts of the page content programmatically. For example, it can be used to scrape articles from web pages, clean up unwanted sections like navigation or footers, resolve relative links based on a base URL, and optionally include media data such as images or videos.
Practical examples:
- Extracting the main article content from a blog post HTML.
- Filtering out external links or certain HTML tags before further processing.
- Counting words in the extracted content to ensure minimum content length.
- Including detailed verbose output for debugging or analysis purposes.
Properties
| Name | Meaning |
|---|---|
| HTML Content | The raw HTML content to process. Example: <html><body><h1>Example</h1><p>Content</p></body></html> |
| Base URL | The base URL used to resolve relative links within the HTML content. Default is https://example.com. |
| CSS Selector | A CSS selector string to focus extraction on specific parts of the HTML (e.g., article.content). Leave empty to process the full page. |
| Exclude External Links | Boolean option to exclude external links from the results. |
| Excluded Tags | Comma-separated list of HTML tags to exclude from processing (e.g., nav,footer,aside). |
| Word Count Threshold | Minimum number of words required for content to be included in the output. |
| Include Media Data | Boolean option to include media data such as images and videos in the output. |
| Verbose Response | Boolean option to include detailed data in the output, such as the processed HTML and HTTP status codes. |
Output
The node outputs JSON data containing the processed content extracted from the provided raw HTML. This includes filtered text content based on the CSS selector and excluded tags, resolved links using the base URL, and optionally media data if enabled.
If the "Include Media Data" option is enabled, the output will also contain information about images and videos found within the processed HTML.
If "Verbose Response" is enabled, the output will include additional details such as the full processed HTML snippet and possibly status codes or metadata related to the crawling operation.
Binary data output is not indicated in the source code or properties, so the node primarily outputs structured JSON content.
Dependencies
- Requires an API key credential for the Crawl4AI service to perform crawling operations.
- The node depends on the Crawl4AI API to process and extract data from the raw HTML.
- No other external dependencies are explicitly mentioned.
- Users must configure the API key credential in n8n to authenticate requests to the Crawl4AI service.
Troubleshooting
Common issues:
- Invalid or missing API key credential will cause authentication failures.
- Providing malformed or incomplete HTML content may result in empty or incorrect extraction results.
- Incorrect CSS selectors might lead to no content being extracted.
- Setting too high a word count threshold could filter out all content unintentionally.
- Excluding essential tags might remove important parts of the content.
Error messages:
- Authentication errors typically indicate invalid or missing API credentials; verify and update the API key.
- Parsing errors may occur if the HTML content is not well-formed; validate the HTML input.
- If no content is returned, check the CSS selector and excluded tags settings to ensure they match the intended content.
Links and References
- Crawl4AI Official Website — For more information about the crawling service.
- CSS Selectors Reference — To help craft CSS selectors for content extraction.
- HTML Specification — For understanding HTML structure and tags.
Note: This summary is based solely on static analysis of the provided source code and property definitions without runtime execution.