ScrapeOps icon

ScrapeOps

Interact with ScrapeOps Proxy, Parser and Data APIs

ScrapeOps

AI-generated

Overview

The ScrapeOps node allows users to interact with three different ScrapeOps APIs: Proxy API, Parser API, and Data API. It is designed to facilitate web scraping and data extraction tasks by providing a unified interface to these APIs. Users can select the API type and configure various parameters to perform web scraping, parse HTML content, or retrieve structured data from supported domains like Amazon, eBay, Indeed, Redfin, and Walmart. This node is beneficial for automating data collection from websites, extracting product or job information, and handling anti-bot protections.

Use Case Examples

  1. Scrape a webpage using the Proxy API with custom headers and cookies to bypass anti-bot measures.
  2. Parse HTML content from an Amazon product page using the Parser API to extract structured data.
  3. Retrieve product details from eBay using the Data API by providing the product URL or item ID.

Properties

Name Meaning
API Select which ScrapeOps API to use: Proxy API, Parser API, or Data API.
URL The URL to scrape (required for Proxy API).
Method HTTP method to use for the request (GET or POST) when using Proxy API.
Advanced Options Additional configuration options for Proxy API such as bypass type, country for geo-targeting, custom cookies and headers, device type, file type, and other request optimizations.
Return Type Format in which to return the response from Proxy API: original server response or JSON format.
Domain Domain to parse or select for custom parsing when using Parser API or Data API.

Output

JSON

  • json - The JSON response from the selected ScrapeOps API call, containing scraped or parsed data or error information.

Dependencies

  • ScrapeOps API key credential

Troubleshooting

  • Ensure a valid ScrapeOps API key is provided; errors will occur if the key is missing or invalid.
  • Check that the selected API type matches the input parameters; unsupported API types will cause errors.
  • For Proxy API, verify that required fields like URL and method are correctly set.
  • If using advanced options, ensure JSON fields for custom headers and cookies are properly formatted.
  • If errors occur, the node can continue on fail and provide error messages with suggestions to check credentials and parameters.

Links

Discussion