AI Agent Web Access: Scraping vs MCP

• DIY scraping. You write HTTP calls and selectors for each site, parse the HTML, and maintain that code. The agent consumes the output your parser produces.
• Raw browser automation. The agent gets a browser tool and works against the live page. It reads a screenshot or accessibility tree, decides what to click, and repeats that reasoning on every run.
• Per-site official API. The site publishes documented endpoints. You request access, receive credentials, and integrate against a versioned contract.
• Mapped MCP server. A site is mapped once into typed actions and extraction schemas, then exposed to any MCP-capable agent. In Rindler, four tools cover an integration: start_session opens the site and returns a session_id plus the actions valid on that screen, dispatch_action runs one, extract_content re-reads the screen as structured records, and close_session releases the browser.

A scraper depends on markup. If the site redesigns, renames a class, or ships an A/B test, a selector can miss and break the scraper, often silently. You usually find out from empty downstream fields. Raw browser automation avoids that specific selector failure by re-reading the page on every run, but it becomes nondeterministic. The same task can take a different path and a different number of steps each time.

An official API is the most stable surface of the four. It changes on the provider's schedule, with versioning and deprecation notices.

A mapped MCP server keeps that volatility behind the mapping. Rindler returns the same response shape for the same call across runs, independent of DOM changes, popups, or A/B-tested layouts. When a site changes its layout, the site configs are self-healing and adapt, and an automated verifier keeps confirming that each mapping still returns reliable, structured data. Your agent code never references a selector, so there is nothing on your side to fix. Agents never touch raw HTML or CSS selectors.

Raw pages are expensive for an agent to process. A direct fetch of a typical listings, search, or dashboard page runs 50,000+ tokens of navigation, footer, sidebar, and tracking markup around the data you need. Screenshot- or tree-driven automation re-reads the page on every step of every run.

A working scraper is cheap on token use. Parsing happens in your code, so the agent sees clean output. The cost moves to breakage and maintenance.

Rindler responses are structured records: a few hundred tokens of JSON, data.listings for a list, data.listing for a detail page, instead of the 50k-token DOM returned by a raw fetch. Every screen exposes structured records through an extraction schema. A raw accessibility tree is the fallback only when no schema is registered. A site's filter and facet controls are read live from the current page as a filters array rather than hardcoded. Linked PDFs are read per page through extract_document, with a query param to narrow long documents before reading. On one live grocery benchmark, the mapped version of the task finished in 98 seconds instead of 407 and cost $0.26 instead of $1.54.

Scraping and raw browser automation leave auth to you: hand-exported cookies, credentials stored next to a script, and sessions that die mid-run. Official APIs handle this well with API keys or OAuth, when the API exists and covers your workflow.

Rindler treats sign-in as a one-time handoff. Your agent receives a single-use sign-in link (a capture_url, which expires in 15 minutes). You open it, sign in on the real site inside a managed browser, complete any MFA step, and click "I'm logged in." Rindler never asks for or stores your password. It keeps the resulting session encrypted, scoped to you and that one site, and loads it automatically on later runs. When a session eventually dies, the next call returns auth_status: "expired" plus a fresh sign-in link instead of failing silently. manage_cookies lists every saved login and can revoke any of them. It returns metadata only, never credential or cookie values. Bot evasion is managed for you.

In a scraper or raw browser session, a purchase click and a pagination click are both just clicks. You have to build any approval policy yourself. Official APIs handle this better because writes are usually separate, explicit endpoints.

A mapped MCP server can make that boundary machine-readable. Rindler enumerates the valid actions per screen. The agent may only dispatch names from that list, and may not invent actions, construct URLs, or reference CSS selectors. The tools also carry MCP annotations your runtime can gate on: extract_content is annotated read-only, while dispatch_action is annotated destructive because it is the one tool that changes external site state (it can submit forms, orders, messages, and applications). That split lets an agent runtime require human approval for the calls that write. Guardrails also run server-side: the dismiss_dialog action refuses buttons that spend money or remove items, carts hand off so you check out on the site yourself, and nothing books, pays, or reorders without your explicit instruction.

An official API has the slowest start when access is gated: keys, review, quotas, and client code. Scraping prototypes quickly, then accumulates maintenance. Raw automation starts fastest because there is nothing to build, and it moves the cost into every run.

Rindler setup is one command, followed by an OAuth 2.0 PKCE approval in your browser. No API key, no client library.

curl https://rindler.ai/install | sh

If the site is in the catalog, your first integration is a prompt. If it is not, you can map it yourself: a Fast crawl takes a minute or two, a Deep crawl several minutes, and the result is a private API for that site, scoped to your account and usable immediately. Multi-tenant platforms compound this: a single mapping of an ATS platform covers every employer on that platform. When a site genuinely is not supported, start_session says so explicitly with an unsupported_site screen. That is a signal to fall back to native browsing, not an error to debug.

• Use the official API when one exists, you can get access, and it covers the reads and writes you need. It is the site's own supported contract, and nothing beats that. Rindler is aimed at the opposite case: sites without a reliable API, such as legacy portals, private SaaS dashboards, government systems, and multi-step forms.
• Use plain scraping or a plain fetch for one-shot reads of a single public page. Rindler is not relevant to one-shot tasks that only need human-readable copy from a single marketing page, and a static page whose maintenance you accept is a fine scraping target.
• Use raw browser automation for exploratory, one-off tasks on sites nobody has mapped. Rindler's own guidance to agents is to use native browsing when start_session returns unsupported_site.
• Use a mapped MCP server when the task repeats, when it writes, or when the site fights you: many Rindler-connected sites answer a direct fetch with a 403, a JS-only shell, or a bot-challenge page, and the MCP path is the supported channel there.

Read how it works (/how-it-works) for the mapping and verification pipeline behind a site config. Use the quickstart (/docs/quickstart) to connect your agent with one command. If the site you need is not in the catalog yet, you can map it yourself (/docs/mapping).