This page documents the purpose and the expected behavior of RankAuthorityBot, the crawler operated by Rank Authority, Inc. It is written for website owners, hosting providers, and security vendors who see our requests in their logs and want to know who we are, why we are there, and how to control us.
The short version: we only crawl sites that someone has connected to a Rank Authority account, our requests are cryptographically signed so you can verify them, and disconnecting a site in the Rank Authority dashboard stops all crawling of it immediately.
01
Overview
What the Crawler Does
Rank Authority is an SEO software platform. When a customer connects their website to our platform, RankAuthorityBot fetches the pages of that site so the platform can analyze them and report what should be improved.
On a connected site, the typical pattern of activity is:
- An initial audit crawl of pages discovered from the site's sitemap and internal links
- Periodic re-checks of those same pages, so recommendations stay current as content changes
- A re-fetch of an individual page shortly after it is published or updated through our platform
02
Scope
Which Sites It Crawls
RankAuthorityBot crawls only sites that have been added to a Rank Authority account. It is not a general-purpose web crawler and it has no queue of sites discovered from the open web.
Specifically, RankAuthorityBot does not:
- Crawl the web at large or follow links off the customer's own site
- Build a search index or a public content archive
- Collect content to train or fine-tune AI models
- Resell, republish, or redistribute the content of the pages it fetches
03
Verification
Verifying Our Requests
Our requests are cryptographically signed using Web Bot Auth (HTTP Message Signatures, RFC 9421). Each request carries a Signature, a Signature-Input, and a Signature-Agent header that points at our public key directory:
https://backend.rankauthority.com/.well-known/http-message-signatures-directory
The directory publishes our Ed25519 public keys as a JSON Web Key Set, and the response itself is signed, so nobody can mirror it and claim to be us. Verifying that signature is the authoritative way to confirm a request really came from Rank Authority: an IP address or a User-Agent string can be copied by anyone, but a valid signature cannot be forged.
04
Identity
User-Agent and robots.txt Token
In robots.txt, address us by the token:
RankAuthorityBot
We want to be straightforward about one detail. Our crawler sends a standard desktop browser User-Agent string rather than a custom bot string, because a large share of sites serve different markup, or refuse to respond at all, to clients that do not look like a browser — and an audit is only useful to the site owner if we see the same HTML their visitors see.
Because of that, please do not use our User-Agent to identify us. Use the request signature described in section 03. The RankAuthorityBot token above is honored for robots.txt matching regardless of what the User-Agent header says.
05
Rate
Request Rate
Requests to a single site are deliberately staggered — by default roughly one page every two seconds. We do not open large parallel bursts against one host.
The rate is also adaptive. If a site starts refusing our requests, the next cycle automatically slows down, up to 30 seconds between requests, and only returns to the default pace after a clean pass. A site that rate-limits us with a 429 response and a Retry-After header will have that delay honored rather than retried against.
06
Controls
Controlling RankAuthorityBot
The most direct control is the Rank Authority dashboard itself: disconnecting a site stops all crawling of it immediately. Beyond that, we read robots.txt on every audit cycle.
User-agent: RankAuthorityBot
Disallow: /members/
Crawl-delay: 5
Crawl-delay is honored, up to a maximum of 10 seconds. A larger value is clamped to 10 seconds, because a misconfigured delay of an hour would stall an audit that the site owner explicitly asked us to run.
Disallow is honored for sites we crawl over public HTTP. There is one exception, and we would rather state it plainly than have you discover it in your logs:
When a site's CMS is connected to Rank Authority with authenticated credentials, the account holder has proven that they control the site and has explicitly instructed us to audit it. In that case Disallow rules do not stop the audit of their own pages. Disconnecting the site, or removing the CMS connection, ends it.
If robots.txt cannot be reached, or the server answers with an error, we continue with the audit. We are crawling at the owner's request rather than indexing a site without permission, so a temporary server error should not silently cancel work they are waiting on.
07
Contact
Contact and Reporting
If our crawler is causing a problem on your site, or you believe you are seeing traffic that claims to be us but is not, contact us and we will investigate. Please include the date, the requested URLs, and the signature headers if you have them.
