SecureScout crawler

How it identifies itself

User-Agent string:

SecureScoutBot/1.0 (+https://securescout.si/bot)

robots.txt token: SecureScoutBot. Rules are matched case-insensitively on the product token (the part before the "/"). If no group names our token, we follow the rules for User-agent: *.

Rules written for the legacy SecureScout token are still honored. When groups for both tokens match, the most restrictive group wins.

How it crawls

• At most one request at a time to any host.
• A minimum delay between requests to each source, currently 1 to 30 seconds depending on the source, and never more than one request per second.
• A Crawl-delay in robots.txt is honored. It can only make us slower, never faster.
• Retries happen only after timeouts, 429 and 5xx responses, with capped exponential backoff, and Retry-After is honored.
• Conditional requests (ETag and Last-Modified), so unchanged pages aren't downloaded again.
• HTTPS only.

How to opt out

To opt out, add a group for the SecureScoutBot token to your robots.txt. This blocks the crawler from your whole site:

User-agent: SecureScoutBot
Disallow: /

To block only part of your site, list those paths instead of "/".

What we honor today

• We follow robots.txt as specified in RFC 9309, including wildcards. We cache robots.txt for up to 24 hours, so a change can take up to a day to apply.
• If robots.txt answers 401, 403, 405, 418, 429 or 451, or with a firewall or challenge page, we treat the whole site as disallowed. One exception: for an official government news feed whose robots.txt is behind a firewall, we fetch only the feed URL we reviewed, never article pages.
• If your robots.txt disallows a path for GPTBot, ClaudeBot, Google-Extended or CCBot, in a group that names that crawler, we treat the path as disallowed for us too.
• A Content-Signal line in robots.txt with search=no, ai-input=no or ai-train=no, in the group that applies to us (ours, or User-agent: * if there is none) or outside any group, opts your whole site out.
• For official data APIs that we use under their published terms, we follow those terms instead of robots.txt.
• We never retry a blocked request with a different user agent, proxy or headers.
• When we crawl HTML pages from a site's sitemap, we drop any page marked noindex, noai or noimageai in an X-Robots-Tag header or a robots meta tag that applies to all crawlers, to SecureScoutBot, or to GPTBot, ClaudeBot, Google-Extended or CCBot.

Not live yet In development

These aren't live yet:

• No feed excerpts for sites that have opted out.

Verifying our crawler

We don't publish IP ranges or reverse-DNS names for the crawler. To check a request, look for the user agent above.