JigsawStack icon

JigsawStack

Use JigsawStack API

Actions20

AI Scrape → AI Scrape

AI-generated

Overview

The AI Scrape operation in the JigsawStack node allows users to scrape web content either from a URL or directly from provided HTML content. It supports extracting specific elements from the page based on user-defined prompts, such as titles, points, or prices. The operation is useful for scenarios where structured data extraction from web pages is needed, such as market research, price comparison, content aggregation, or data mining from websites.

Use Case Examples

  1. Scraping product prices and titles from an e-commerce page by providing the URL and element prompts.
  2. Extracting specific sections of a webpage by supplying the HTML content and defining CSS selectors and element prompts.

Properties

Name Meaning
Scrape Source Choose the source of the scrape, either from a URL or raw HTML content.
URL The URL of the page to scrape, required if Scrape Source is set to URL.
HTML The raw HTML content to scrape, required if Scrape Source is set to HTML.
Element Prompts Array of element prompts to extract from the page, such as titles, points, or prices.
Root Element Selector CSS selector to limit the scope of scraping to a specific element and its children.
Page Position Current page number for pagination, minimum value is 1.
Advance Config Advanced configuration options for the scraper including console, network, cookies, HTTP headers, reject request patterns, and goto options.
Http Headers Custom HTTP headers to send with requests as key-value pairs.
Reject Request Pattern Array of patterns to intercept and block requests, such as image file types.
Goto Options Custom page-load behavior settings.
Wait For Wait condition before scraping, can be a timeout, selector, or function.
Cookies Cookies to set for the page request.
Size Preset Predefined screen size preset, e.g., HD, FHD, 4K UHD.
Is Mobile Whether to emulate a mobile device viewport.
Scale Device scale factor, minimum value is 1.
Width Viewport width in pixels.
Height Viewport height in pixels.
Force Rotate Proxy Whether to force proxy rotation for each request, which may incur additional costs.
BYO Proxy Bring-your-own-proxy configuration including server URL and optional authentication credentials.
Request Options Node-level request options such as batching, ignoring SSL issues, proxy settings, and timeout.

Output

JSON

  • data - The scraped data extracted from the webpage or HTML content based on the element prompts.

Dependencies

  • JigsawStack API key credential

Troubleshooting

  • Ensure either URL or HTML is provided but not both, as the operation requires one source of content to scrape.
  • Verify that the element prompts are correctly specified to extract the desired data elements.
  • Check network connectivity and API key validity for accessing the JigsawStack API.
  • If using custom HTTP headers, cookies, or proxy settings, ensure they are correctly configured to avoid request failures.
  • Pagination settings like Page Position should be set correctly to scrape multi-page content.

Links

Discussion