About the RDA-ContentIntelligence crawler

RDA-ContentIntelligence fetches public pages when someone requests a Content Intelligence Index from RDA Digital for a website. It runs only on request, never on a schedule.

• It stays on the domain it was asked about. A report for example.com reads example.com and its subdomains, such as www.example.com and blog.example.com, and nothing else: it doesn't follow redirects, sitemaps, or language links to other domains. If the homepage redirects to another domain, the report stops there.
• Each report reads the homepage (also over http and with or without www), robots.txt, llms.txt, and XML sitemaps. It then reads up to 19 more pages, up to 3 language versions of the homepage, and checks up to 30 internal links for errors.
• At most 5 requests are in flight at a time, and a report downloads at most 40 MB in total.
• The pages, language versions, and link checks follow robots.txt rules for its own token and for *, including the robots.txt of each subdomain it reads.
• It does not run JavaScript, submit forms, log in, or store copies of your pages.
• It requests only public http(s) URLs.

To stop it from reading your pages, add this to robots.txt:

User-agent: RDA-ContentIntelligence
Disallow: /

Rules written for our earlier name, RDA-ContentConfidence, still apply.

When someone requests a report for your site, it still reads the homepage, robots.txt, llms.txt, and sitemaps, but nothing else.

Questions or concerns: contact RDA Digital.