Chimera v0.10.0 — map + crawl (whole-site scraping)

Phase 2 of the web verb set — whole-site map and crawl complete the scrape / extract / map / crawl quartet. All built in, robots-aware, no new dependency.

Added

• map — list a site's URLs cheaply. Reads the sitemap (robots.txt Sitemap: lines + /sitemap.xml, one index level deep), falling back to scanning the page's same-domain links. Optional search keyword filter. Run it to scope a site before crawling.
• crawl — BFS across a site. Follows links from a seed URL and returns each page's clean Markdown, bounded by limit + max_depth, same-domain by default, deduped, with include/exclude URL globs. robots.txt-aware: obeys Disallow and Crawl-delay by default (opt out with respect_robots=false). Reuses the Phase-1 cost-aware fetch cascade.

Both tools are data-fenced and taint the run (untrusted web content). +7 tests.

With this, the full quartet — scrape / extract / map / crawl — ships in the base: fetch and read any page, pull fields as injection-safe JSON, list a site, and crawl it. An agent-native scraper, no external service required.

pip install --upgrade chimera-agent