Smart Crawler
AI-generatedSummary
Extract data from a web page by selecting a list of items using a CSS selector and then mapping specified fields from each item, optionally following links to secondary pages to gather additional data.
Inputs
- url (required) — The URL of the web page to crawl.
- cookie — Optional HTTP Cookie string to include with the requests.
- listSelector (required) — CSS selector to identify the list items containing the data to extract.
- useBrowser — Whether to use Puppeteer to render the page for JavaScript-heavy sites.
- fields (required) — Configuration of fields to extract from each list item, including name, CSS selector, extraction type (text, HTML, attribute), and field type (normal or jump). For jump fields, additional parameters define the link selector, target page selector, and subfields to extract on the target page.
Output shape
a list of items, each corresponding to one matched element from the listSelector, containing extracted fields as JSON properties, including nested objects for jump fields.
The operation may return multiple output items per input item, each paired with its input index. If useBrowser is enabled, Puppeteer renders the pages. Errors during extraction either stop execution or return an error field in the output item depending on the node's continueOnFail setting.
Examples
Example 1: Extract product names and prices from a product list page with clickable links to detail pages that contain descriptions.
url: 'https://example.com/products', listSelector: '.product-item', useBrowser:false, fields: [{name:'productName', selector:'.title', type:'text', fieldType:'normal'}, {name:'price', selector:'.price', type:'text', fieldType:'normal'}, {name:'details', selector:'.title', fieldType:'jump', clickSelector:'a.detail-link', targetSelector:'.product-description', jumpFields:[{name:'description', selector:'.desc', type:'text'}]}]