DEV Community

Cover image for I built an open-source browser agent that turns any website into structured JSON
Krishna shakula
Krishna shakula

Posted on

I built an open-source browser agent that turns any website into structured JSON

#opensource #webscraping #ai #python

I built an open-source browser agent that turns any website into structured JSON

Most web scraping tools still think the internet is made of clean HTML.

It is not.

The modern web is made of:

  • JavaScript-heavy pages
  • login-ish flows
  • dropdowns
  • old government forms
  • pricing pages with no API
  • websites that change their DOM every week
  • pages where the useful data is behind clicks, tabs, filters, and forms

So I built Browsewright:

Give an LLM a URL and a goal. It drives a real Chrome browser, completes the task, and returns structured data instead of raw HTML.

GitHub: https://github.com/krishnashakula/browsewright

pip install browsewright
Enter fullscreen mode Exit fullscreen mode

Then run:

bw "https://stripe.com" "what does this company do and who is it for"
Enter fullscreen mode Exit fullscreen mode

Instead of writing selectors, XPath, browser scripts, retry logic, and extraction code, you give it intent.

Browsewright figures out the path.

The idea

Playwright automates a browser you script.

Browsewright is the browser that scripts itself.

With traditional browser automation, you write code like:

page.goto(url)
page.click("#pricing")
page.wait_for_selector(".plans")
page.locator(".price-card").all()
Enter fullscreen mode Exit fullscreen mode

That works until the website changes.

Browsewright flips the model.

You say:

bw "https://example.com/pricing" "extract all plans, prices, limits, and billing options as JSON"
Enter fullscreen mode Exit fullscreen mode

The LLM reads the page, decides what matters, drives the browser only when needed, and returns structured output.

Why I built it

I kept seeing the same problem in AI agents, lead enrichment systems, market research tools, and n8n-style automations:

The LLM is smart enough to understand the task.

But it cannot reliably operate the web.

Most systems stop at one of these layers:

  1. Fetch raw HTML
  2. Convert the page to Markdown
  3. Ask the LLM to summarize it
  4. Hope the answer is useful

That is fine for simple pages.

But real business workflows need action:

  • Fill this form
  • Find pricing
  • Search this registry
  • Compare competitors
  • Track stock availability
  • Extract lead data
  • Monitor brand mentions
  • Submit a query
  • Return clean JSON

That is not just scraping.

That is browser work.

What Browsewright does

Browsewright is an open-source Python tool that lets an LLM operate a real browser.

It can:

  • read a page
  • decide what to click
  • fill forms
  • submit forms
  • extract structured JSON
  • use cheaper shortcuts before launching Chrome
  • run locally with your own API key
  • expose itself as an MCP server for AI assistants

Install:

pip install browsewright
Enter fullscreen mode Exit fullscreen mode

Use it from the CLI:

bw "https://news.ycombinator.com" "what is the top story right now"
Enter fullscreen mode Exit fullscreen mode

Use JSON mode:

bw "https://example.com" "find the pricing plans" --json
Enter fullscreen mode Exit fullscreen mode

Debug visually:

bw "https://example.com" "debug this page" --no-headless --verbose
Enter fullscreen mode Exit fullscreen mode

Use it in Python:

import asyncio
from browsewright import search

res = asyncio.run(
    search(
        "https://stripe.com",
        "what does this company do and who is it for?"
    )
)

print(res.answer)
print(res.stage)
print(res.tokens_total)
print(res.elapsed_s)
Enter fullscreen mode Exit fullscreen mode

It does not just read the web. It acts on the web.

Most AI scraping tools can summarize a page.

Browsewright can operate a page.

Example:

bw-tasks form \
  "https://registers.maryland.gov/RowNetWeb/Estates/frmEstateSearch2.aspx" \
  --profile examples/sample_profile.json
Enter fullscreen mode Exit fullscreen mode

That is not a clean JSON API.

That is not a modern developer-friendly endpoint.

It is an old form-based website.

Browsewright can inspect the form, understand the fields, map a profile to the inputs, choose valid dropdown options, submit the form, and extract results.

No selectors.

No XPath.

No custom script for that one website.

Just intent.

The cheap path comes first

One design goal was simple:

Do not launch a browser unless you actually need a browser.

A lot of sites can be answered from cheaper paths:

  • public APIs
  • RSS feeds
  • JSON endpoints
  • public archives
  • static content

So Browsewright tries a pre-flight pipeline first.

Only if that fails does it spin up Chrome.

That makes it useful for workflows where cost matters.

In my benchmark, Browsewright extracted data from 50 real websites for about 4.7 cents total.

Reproduce it:

python examples/batch_test.py
Enter fullscreen mode Exit fullscreen mode

Built-in workflows

Browsewright ships with task commands for common business automation use cases.

1. Competitor watch 

bw-tasks watch "https://competitor.com/pricing"
Enter fullscreen mode Exit fullscreen mode

Use this to monitor pricing pages, feature pages, changelogs, positioning, and landing pages.

2. Lead enrichment 

bw-tasks enrich "https://company.com"
Enter fullscreen mode Exit fullscreen mode

Returns CRM-style fields and a personalized first line for outreach.

Example output:

{
  "company_name": "ExampleCo",
  "industry": "AI SaaS",
  "icp_fit_score_1_to_10": 8,
  "personalized_cold_email_first_line": "I noticed your team is expanding into enterprise AI workflows..."
}
Enter fullscreen mode Exit fullscreen mode

3. Agentic form fill 

bw-tasks form "https://example.com/form" --profile profile.json
Enter fullscreen mode Exit fullscreen mode

Useful for authorized internal workflows, testing, research forms, and repetitive manual data-entry tasks.

4. Price and stock tracking 

bw-tasks track "https://store.com/product"
Enter fullscreen mode Exit fullscreen mode

Track product availability, price movement, and page changes.

5. Brand monitoring 

bw-tasks brand "Browsewright" "https://site1.com" "https://site2.com"
Enter fullscreen mode Exit fullscreen mode

Collect mentions, sentiment, and summaries across selected URLs.

Works with MCP

Browsewright can also run as an MCP tool.

Install with MCP support:

pip install "browsewright[mcp]"
Enter fullscreen mode Exit fullscreen mode

Add it to your MCP client:

{
  "mcpServers": {
    "browsewright": {
      "command": "bw-mcp"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Now your AI assistant gets a tool like:

read_page(url, goal)
Enter fullscreen mode Exit fullscreen mode

That means an AI assistant can research the web with a real browser instead of only guessing from static text.

Why structured JSON matters

A summary is nice.

JSON is useful.

For automation, you usually need data that can go into another system:

  • a CRM
  • a spreadsheet
  • a database
  • Slack
  • n8n
  • Make
  • Zapier
  • a vector database
  • an internal dashboard
  • another AI agent

Browsewright has a core structured extraction primitive:

import asyncio
from browsewright import extract_structured

schema = {
    "headline": "string",
    "open_roles": [
        {
            "title": "string",
            "team": "string",
            "location": "string"
        }
    ]
}

data = asyncio.run(
    extract_structured(
        "https://example.com/careers",
        schema,
        instruction="Extract the page headline and every open job posting."
    )
)

print(data["open_roles"])
Enter fullscreen mode Exit fullscreen mode

This is the part I care about most:

Not “AI browsed a website.”

More like:

AI turned a messy website into a reliable automation input.

Where this is useful

Browsewright is useful when you need web data but the site does not give you a clean API.

Examples:

AI agents

Give your agent the ability to inspect pages, fill forms, and return structured output.

n8n workflows

Use Browsewright as the browser/research layer, then pass JSON into the rest of your workflow.

Sales automation

Enrich leads from company websites, pricing pages, career pages, and recent announcements.

Market research

Extract competitor positioning, product pages, integrations, pricing, and messaging.

Recruiting

Extract job listings from company career pages into clean structured data.

Monitoring

Track changes on important URLs and alert when something changes.

Internal operations

Automate repetitive browser tasks on systems where you are authorized to operate.

A simple n8n-style flow

Here is the kind of workflow Browsewright is designed for:

Schedule trigger
   ↓
Run Browsewright on competitor pricing page
   ↓
Extract structured JSON
   ↓
Compare with previous snapshot
   ↓
If changed, send Slack alert
   ↓
Save result to database
Enter fullscreen mode Exit fullscreen mode

The browser is only one piece.

The output is what makes the automation useful.

Important note on responsible use

Browsewright is designed for authorized research, your own properties, internal automation, and websites where automated access is allowed.

Use polite mode.

Respect robots.txt.

Respect rate limits.

Respect terms of service.

Do not use browser automation to abuse websites, bypass rules, collect private data, or violate laws.

The best use case is not “scrape everything.”

The best use case is:

Replace repetitive manual browser work with reliable, transparent, authorized automation.

How it compares conceptually

Tool type Good at Weakness
HTTP scraper Fast static extraction Breaks on JS, forms, interaction
Browser script Precise automation Requires selectors and maintenance
Search API Quick answers Limited action, limited page control
LLM summarizer Natural language answers Often not structured or repeatable
Browsewright Intent → browser action → JSON Best for tasks where browser interaction matters

Browsewright is not trying to replace every scraper.

It is for the cases where plain scraping is too brittle and full manual browser scripting is too much work.

Try it

Install:

pip install browsewright
Enter fullscreen mode Exit fullscreen mode

Run:

bw "https://example.com" "find the pricing and return the plans as JSON"
Enter fullscreen mode Exit fullscreen mode

Clone:

git clone https://github.com/krishnashakula/browsewright
cd browsewright
pip install -e .
Enter fullscreen mode Exit fullscreen mode

Use MCP:

pip install "browsewright[mcp]"
Enter fullscreen mode Exit fullscreen mode

GitHub:

https://github.com/krishnashakula/browsewright

What I want feedback on

I am looking for feedback from developers building:

  • AI agents
  • scraping systems
  • research agents
  • n8n workflows
  • sales automation
  • market intelligence tools
  • browser automation tools
  • MCP servers

Especially:

  1. What websites should I test next?
  2. What built-in tasks should Browsewright support?
  3. What MCP workflows would make this more useful?
  4. What would make you trust browser-agent output in production?
  5. What should the API look like for teams using this at scale?

If this saves you from writing one more fragile scraper, please star the repo.

GitHub: https://github.com/krishnashakula/browsewright

The web was built for humans.

Browsewright lets agents use it like humans — and return data like machines.

Top comments (1)

Collapse
 
bhavin-allinonetools profile image
Bhavin Sheth •

Really like the "intent over selectors" approach. I've spent way too much time fixing broken scrapers after site updates, so returning structured JSON from real browser actions feels much closer to how these workflows actually work.