CSS Selector Extractor
AI-generatedOverview
The node "Crawl4AI: Content Extractor" is designed to extract structured content from web pages by using CSS selectors. It is particularly useful for scraping data from websites where the content is organized in repeating elements such as product listings, article cards, or any other structured blocks. Users can specify a base CSS selector that identifies the repeating element and then define multiple fields with their own relative CSS selectors to extract specific pieces of information like text, HTML, or attribute values.
Practical examples include:
- Extracting product details (title, price, image URL) from an e-commerce category page.
- Scraping article headlines, authors, and publication dates from a news site.
- Collecting event information (name, date, location) from an event listing page.
The node supports browser automation features such as enabling JavaScript execution, running in headless mode, executing custom JavaScript code before extraction (e.g., clicking "load more" buttons), and setting viewport dimensions, which helps in handling dynamic content loading.
Properties
| Name | Meaning |
|---|---|
| URL | The web page URL to extract content from. |
| Base Selector | CSS selector identifying the repeating element on the page (e.g., each product item or article card). |
| Fields | A collection of fields to extract from each repeating element. Each field includes: - Field Name: Identifier for the extracted data. - CSS Selector: Relative selector within the base element. - Field Type: Type of data to extract (Text, HTML, Attribute). - Attribute Name: If extracting an attribute, specify its name (default "href"). |
| Browser Options | Settings controlling browser behavior: - Enable JavaScript: Whether to run JS on the page. - Headless Mode: Run browser without UI. - JavaScript Code: Custom JS to execute before extraction. - Timeout (ms): Max wait time for page load. - Viewport Height & Width: Browser window size. |
| Options | Additional options: - Cache Mode: How to use caching (enabled, bypass, read-only). - Include Original Text: Whether to include full original webpage text in output. - Clean Text: Normalize extracted text by removing extra spaces and newlines. |
Output
The node outputs JSON data representing an array of objects, each corresponding to one instance of the base selector found on the page. Each object contains key-value pairs where keys are the field names defined in the input properties, and values are the extracted content according to the specified field type (text, HTML, or attribute value).
If enabled, the output may also include the full original text of the webpage.
The node does not explicitly mention binary data output, so it is assumed to produce only JSON structured data.
Dependencies
- Requires an API key credential for the Crawl4AI service to perform content extraction.
- Uses a headless browser environment to load and interact with web pages, including JavaScript execution if enabled.
- Network access to the target URLs is necessary.
- No additional environment variables or n8n configurations beyond the API credential are indicated.
Troubleshooting
Common Issues:
- Incorrect CSS selectors leading to empty or incomplete extraction results.
- Pages requiring JavaScript to render content might need "Enable JavaScript" option turned on.
- Dynamic content loading may require custom JavaScript code to trigger loading (e.g., clicking "load more" buttons).
- Timeout errors if the page takes too long to load; increase the timeout setting accordingly.
- Cache mode misconfiguration could cause stale data or failure to fetch fresh content.
Error Messages:
- Authentication errors related to the API key: Verify the API credential is correctly configured.
- Network errors when accessing the URL: Check network connectivity and URL validity.
- Extraction failures due to invalid selectors: Review and test CSS selectors carefully.
- JavaScript execution errors: Validate custom JS code syntax and logic.