SlidefactoryInstall

Summary

Scrape the contents from a specified list of URLs using a chosen scraping provider.

Inputs

• URLs (required) — Provide a comma-separated list of URLs to scrape content from. The string is converted into an array internally.
• Scraping Provider — Select the scraping provider to use for extracting content, options are 'Jina' and 'Firecrawl' (default is 'Jina').
• Use Cache — Boolean flag to determine whether to use cached results if available (default true).
• Additional Options — Configure options including minimum content length (default 80), maximum content size (default -1 for unlimited), and output format ('markdown' or 'text', default 'markdown').
• Request Options — Advanced request settings such as batching, SSL verification bypass, proxy usage, and request timeout in milliseconds (default 10000).

Output shape

A JSON object containing the scraped content extracted from the requested URLs.

The URLs input is split into an array by commas and whitespace. The operation sends a POST request to '/api/scraping/scrape' with relevant parameters. The output format can be controlled by 'Additional Options'. Caching behavior is configurable via 'Use Cache'. No mention of pagination or multiple response pages; output is the scrape result for the provided URLs.

Examples

Example 1: Scrape contents of two URLs with default scraping provider and markdown format

Set URLs to 'https://example.com, https://example2.com', leave default scraping provider (Jina), and defaults for other options.