ROILadderBot
The crawler identified as ROILadderBot/1.0 (+https://roiladder.com/bot).
If you found this page in your server logs, that string is mine.
What it does
It reads a small number of publicly available pages on business websites to understand what a business does and how its site is built. It is looking at page structure, headings, metadata, and whether a site works on a phone.
How it behaves
- Respects
robots.txt. - One request every two seconds at most, to one site at a time.
- Backs off immediately on a 429 or a 503.
- Reads a handful of pages per site, not the whole thing.
- Never logs in, never creates an account, never solves a CAPTCHA, never uses a proxy to get around a block.
- Identifies itself honestly in every request, which is why you are reading this.
Keeping it out
Add this to your robots.txt:
User-agent: ROILadderBot
Disallow: /
It will be honored on the next visit. Or email gus@roiladder.com and I will exclude your domain by hand, permanently, no questions.
Operated by ROILadder. Contact: gus@roiladder.com