Spectrawl web layer for agents

Developers building research or monitoring tools that need current web content for an agent.

Use it now Want it running at your company?

Recommendedour rating
28gitHub stars
MITlicence
7 months agolast update

Use it now

git clone https://github.com/Pyx-Corp/spectrawl.git

Open the original ↗

Third party software. By copying, downloading or installing it you accept the Academy legal notice: you are responsible for reviewing its code and permissions before use.

README.md

Loading the file...

What it does

A self-hosted package that gives AI agents web search, page reading, multi-page crawling and structured extraction. Search runs through Gemini Grounded Search, then the top pages are fetched in full so the agent's model can read the sources itself. It also includes a stealth browser, cookie-based sign-in and adapters for several platforms.

Use cases

01Searching the web and reading the top sources
02Crawling a documentation site for its content
03Extracting fields from a product page

Questions about
Spectrawl web layer for agents.

What is Spectrawl web layer for agents used for?

Developers building research or monitoring tools that need current web content for an agent. A self-hosted package that gives AI agents web search, page reading, multi-page crawling and structured extraction. Search runs through Gemini Grounded Search, then the top pages are fetched in full so the agent's model can read the sources itself. It also includes a stealth browser, cookie-based sign-in and adapters for several platforms.

How do I install Spectrawl web layer for agents?

Run this in your terminal: git clone https://github.com/Pyx-Corp/spectrawl.git

Is Spectrawl web layer for agents open source?

Yes. The code is on GitHub (Pyx-Corp/spectrawl) under the MIT licence.

Is Spectrawl web layer for agents safe to use?

It passed our automatic scan for credential theft, hidden instructions and risky install commands. Third party open source software. Sabemos AI does not maintain it. Check the code and permissions before you connect it to company data.

Related resources

Read the guide: Private AI on your company documents ↗