SnagaBot
The crawler of Snaga, an independent web search for AI agents.
If you see SnagaBot in your logs, it is us: Snaga, run by Squad AI Inc. We build our own index of the web so that AI agents can find pages. Each result links to your page and shows what your page says about itself. We don't sell your content: agents follow the link to read it.
User agent
Mozilla/5.0 (compatible; SnagaBot/0.1.0; +https://snaga.ai/bot)What it fetches
| When | What it fetches and how often |
|---|---|
| Crawling, to keep our index | /robots.txt, then HTML pages it allows; other files (PDFs, images, video) are skipped. One request at a time per host, at least 2 seconds apart; your Crawl-delay is used when it's longer. |
| Fetching a page an agent asked for | That one page, after checking your /robots.txt. At most one request per second per host; a page is fetched again no sooner than 15 minutes later. |
We don't log in, accept cookies, fill in forms, solve CAPTCHAs or get around paywalls or blocks. We don't read more than 2 MB of a page when crawling, or 5 MB when an agent asks for it.
Rules it follows
- robots.txt as RFC 9309 defines it: the group for snagabot if you have one, otherwise the one for *. If your robots.txt returns a server error, we fetch nothing from the host until it can be read. We read it again every day.
- noindex (robots meta tag or X-Robots-Tag): the page is not kept.
- nofollow, and links marked rel="nofollow", "ugc" or "sponsored": not followed or counted.
- nosnippet and max-snippet:N: we show no text of the page, or at most N characters.
- Slowing down: if your server takes longer to answer, we wait ten times as long between requests; on 429 or 503 with Retry-After we wait that long; a host that asks for more than an hour is left alone for the rest of the crawl.
Directives addressed to snagabot apply to us alone; directives for other crawlers, such as googlebot, don't. For example:
<meta name="snagabot" content="noindex">
X-Robots-Tag: snagabot: nosnippetHow to limit or block it
To keep SnagaBot off the whole site:
User-agent: snagabot
Disallow: /For part of a site, or to slow it down:
User-agent: snagabot
Disallow: /private/
Crawl-delay: 10Part of our index also comes from Common Crawl. Blocking CCBot keeps your pages out of their future crawls, and so out of ours.
Removal and contact
To have pages or a whole domain removed from our index, or never crawled, write to us. We keep a list of domains we never fetch, and act on requests promptly.
Contact: agent@snaga.ai
We don't collect personal data and don't build profiles of people. Text we keep from pages is used to find them, has e-mail addresses and phone numbers removed, and is never served in full.