RitalinBot 0.1 · sitemap reader
RitalinBot 0.1
Active source:your site's sitemaps
Our crawler, RitalinBot, reads websites' sitemaps to track when pages change.What it does and how to block it.
1 request at a time · once a day
Obeys robots.txt (RFC 9309), including Disallow, Allow and Crawl-delay up to 10 s. Waits at least 1 s between requests. Backs off on 429 and 503. Doesn't follow links, fill in forms, run scripts or log in. Responses cut off at 5 MB.
Web crawler