ScrapegraphAI icon

ScrapegraphAI

Turn any webpage into usable data with the ScrapeGraphAI v2 API — scrape, extract, search, crawl, monitor, history, credits.

Actions19

Monitor → Create

AI-generated

Overview

This node operation creates a new monitor for a specified public URL using the ScrapeGraphAI API. It allows users to set up automated monitoring of web pages at defined intervals using cron expressions. The monitor can capture various output formats such as markdown, HTML, images, JSON, links, screenshots, and summaries, with options to customize how the page is fetched and processed. This is useful for tracking changes on websites, extracting structured data, or receiving notifications via webhook on each monitoring tick.

Use Case Examples

  1. Monitoring a product page to track price changes and receive updates every 30 minutes.
  2. Setting up a monitor for a news website homepage to capture and summarize new articles daily.
  3. Creating a monitor for a competitor's website to capture screenshots and HTML snapshots for analysis.

Properties

Name Meaning
URL Public URL to monitor, required to specify the webpage to be tracked.
Name Human-readable name for the monitor to identify it easily.
Interval (Cron) Cron expression defining the schedule for when the monitor runs.
Formats Output formats to capture from the page, such as markdown, HTML, images, JSON, links, screenshots, or summaries, with additional customization options per format type.
Webhook URL Optional URL to receive a POST request on every monitor tick for integration or notification purposes.
Fetch Config Settings controlling how the page is fetched before processing, including cookies, country for geo-targeting, custom headers, fetch mode (auto, fast, JS rendering), scroll count, stealth mode, timeout, and wait time after page load.

Output

JSON

  • id - Unique identifier of the created monitor.
  • name - Name of the monitor.
  • url - URL being monitored.
  • interval - Cron expression defining the monitoring schedule.
  • formats - List of output formats configured for the monitor.
  • webhookUrl - Webhook URL to receive POST requests on each tick, if set.
  • fetchConfig - Configuration details for how the page is fetched before processing.
  • createdAt - Timestamp when the monitor was created.

Dependencies

  • Requires an API key credential for ScrapeGraphAI to authenticate requests.

Troubleshooting

  • Common issues include invalid or malformed cron expressions for the interval, which will cause scheduling errors.
  • Incorrect URL format or inaccessible URLs may cause the monitor creation to fail.
  • Misconfigured fetch options such as invalid JSON in cookies or headers can lead to request failures.
  • If the webhook URL is invalid or unreachable, POST notifications will fail but may not prevent monitor creation.

Links

Discussion