LLM Extractor
AI-generatedOverview
The node "Crawl4AI: Content Extractor" is designed to extract structured content from web pages by leveraging a large language model (LLM) and browser automation. It is particularly useful for scenarios where you want to scrape or extract specific data points from websites without manually coding scrapers. For example, it can be used to extract job listings with titles, locations, and URLs from a careers page, product details from e-commerce sites, or any other structured information embedded in web content.
This node supports flexible schema definitions for the extraction target, allowing users to specify exactly what data to extract and how it should be structured. It also provides options to control browser behavior (like enabling JavaScript, running headless, executing custom JS code), LLM parameters (model choice, temperature, penalties), and caching strategies to optimize repeated extractions.
Properties
| Name | Meaning |
|---|---|
| URL | The web page URL from which to extract content. |
| Extraction Instructions | Text instructions given to the LLM describing what to extract from the page. For multiple items, specify that all items found should be extracted. |
| Schema Type | Method to define the extraction schema: - From Attribute Descriptions: Define fields with names, types, and descriptions. - Generate From JSON Example: Provide an example JSON object to generate schema. - Define using JSON Schema: Manually provide a JSON Schema definition. |
| Schema Fields | When using "From Attribute Descriptions", this defines the list of fields to extract, each with: - Field Name - Field Type (String, Number, Boolean, Array) - Description to guide extraction - Whether field is required |
| JSON Example | When using "Generate From JSON Example", this is the example JSON object that defines the structure to extract. |
| JSON Schema | When using "Define using JSON Schema", this is the JSON Schema string specifying the exact structure and validation rules for the extracted data. |
| Browser Options | Controls browser behavior during extraction: - Enable JavaScript execution - Headless mode - Custom JavaScript code to run before extraction (e.g., click buttons) - Timeout for page load - Viewport size (width & height) |
| LLM Options | Parameters controlling the LLM extraction process: - Frequency Penalty - Max Tokens for response - Model name or ID (optional override) - Presence Penalty - Temperature (randomness) - Top P (nucleus sampling) |
| Options | Additional extraction options: - Cache Mode (enabled, bypass, read-only) - Extract Multiple Items (true/false) - Include Original Text in output - CSS Selector to focus extraction on a part of the page |
Output
The node outputs JSON data representing the structured content extracted from the specified web page according to the defined schema. The output JSON matches the schema or example provided, containing the requested fields and values.
If configured to extract multiple items, the output will typically be an array of objects; otherwise, a single object.
If "Include Original Text" is enabled, the output may also include the full original webpage text.
Binary data output is not indicated in the source or properties, so the node primarily outputs JSON structured data.
Dependencies
- Requires an API key credential for the Crawl4AI service to perform content extraction.
- Uses a headless browser environment to load and interact with web pages.
- Relies on an LLM provider configured via credentials for natural language extraction tasks.
- No explicit environment variables are mentioned, but proper API authentication setup in n8n is necessary.
Troubleshooting
Common Issues:
- Extraction fails if the URL is invalid or inaccessible.
- Incorrect or incomplete extraction instructions may lead to missing or inaccurate data.
- Schema mismatch between expected and actual extracted data can cause errors.
- JavaScript execution disabled on pages requiring dynamic content loading may result in incomplete extraction.
- Timeout too short for slow-loading pages can cause premature failure.
Error Messages:
- Authentication errors indicate missing or invalid API credentials.
- Timeout errors suggest increasing the timeout setting under Browser Options.
- Schema validation errors mean the extracted data does not conform to the defined schema; verify schema correctness and extraction instructions.
- Cache-related errors may occur if cache mode is set to read-only but no cached data exists.
Resolutions:
- Verify URL accessibility and correctness.
- Refine extraction instructions for clarity and completeness.
- Adjust schema definitions or example JSON to match expected output.
- Enable JavaScript and/or add custom JS code to handle dynamic page elements.
- Increase timeout and viewport size as needed.
- Check and update API credentials in n8n settings.