Scrapering MCP Server

Scrapering exposes an MCP (Model Context Protocol) server over HTTP. Connect any compatible AI agent or IDE to give it live web scraping and search capabilities — no custom code required.

Quick Setup

The fastest way to connect Scrapering to your AI agent is with the scrapering CLI:

npx scrapering init

The command detects your environment, writes the correct config file, and prompts for your API key. Supported agents: Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, Codex CLI, opencode, and OpenClaw.

MCP Endpoint

POST https://app.scrapering.com/api/mcp

Available Tools

parse_url

Fetches a URL using a real browser and returns its content, cookies, and an optional screenshot. Mirrors the URL Scraper API.

Requires the url-scraper scope on your API key.

search_google

Searches Google and returns structured SERP results including organic listings, ads, knowledge graph, and optionally an AI overview. Mirrors the Google SERP Scraper API.

Requires the serp:google scope on your API key.

search_yandex

Searches Yandex and returns structured SERP results. Mirrors the Yandex SERP Scraper API.

Requires the serp:yandex scope on your API key.

list_topics

Returns the Google News topic categories (e.g. Business, Technology, Sports) available for a given country. Each topic includes a token needed by get_articles. Mirrors the Google News Topics API.

get_articles

Fetches recent Google News articles for a topic token obtained from list_topics. Mirrors the Google News Scraper API.

Manual Setup

If you prefer to configure manually, add the following to your agent's config file with YOUR_API_KEY replaced by your actual key.

Claude Desktop

~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "scrapering": {
      "url": "https://app.scrapering.com/api/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Cursor

.cursor/mcp.json in your project root or ~/.cursor/mcp.json globally:

{
  "mcpServers": {
    "scrapering": {
      "url": "https://app.scrapering.com/api/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Windsurf

~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "scrapering": {
      "url": "https://app.scrapering.com/api/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

VS Code (GitHub Copilot)

.vscode/mcp.json in your workspace:

{
  "servers": {
    "scrapering": {
      "type": "http",
      "url": "https://app.scrapering.com/api/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Claude Code

claude mcp add --transport http scrapering https://app.scrapering.com/api/mcp \
  --header "Authorization: Bearer YOUR_API_KEY"

Or add to .mcp.json in your project root to share with your team:

{
  "mcpServers": {
    "scrapering": {
      "type": "http",
      "url": "https://app.scrapering.com/api/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

Codex CLI

~/.codex/config.toml:

[mcp_servers.scrapering]
url = "https://app.scrapering.com/api/mcp"
bearer_token_env_var = "SCRAPERING_API_KEY"
enabled = true

bearer_token_env_var is the name of the environment variable holding your key:

export SCRAPERING_API_KEY=YOUR_API_KEY

opencode

~/.config/opencode/opencode.json or ./opencode.json:

{
  "mcp": {
    "scrapering": {
      "type": "remote",
      "url": "https://app.scrapering.com/api/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

OpenClaw

~/.openclaw/openclaw.json:

{
  "mcp": {
    "servers": {
      "scrapering": {
        "url": "https://app.scrapering.com/api/mcp",
        "transport": "streamable-http",
        "headers": {
          "Authorization": "Bearer YOUR_API_KEY"
        }
      }
    }
  }
}

Agent Workflow Examples

Once connected, prompt your agent naturally. It will select and chain the right tools automatically.

Scrape a page and summarize it:

Fetch https://example.com and give me a summary of the main content.

Research a topic with live search:

Search Google for "latest AI regulation news" and summarize the top 5 results.

Compare search results across engines:

Search for "best electric cars 2026" on both Google and Yandex. Compare what's different between the two sets of results.

Monitor competitor pages:

Scrape https://competitor.com/pricing and extract all plan names and prices.

News briefing workflow:

List the top Google News topics for the US, then fetch the 10 latest articles from the Business topic and write a morning briefing.

Deep research with scraping:

Search Google for "open source LLM benchmarks 2026", then scrape the top 3 results and compare the benchmark scores they mention.

Mobile vs desktop comparison:

Fetch https://example.com once as desktop and once as mobile. What differences do you see in the content or layout?

Next steps

• Authentication — manage API keys and assign the required scopes
• URL Scraper — use the REST API directly
• Google SERP Scraper — REST API for Google search
• Google News Scraper — REST API for Google News
• Pricing — each tool call costs the same as the equivalent REST API request