About DirectoriesBot
Last updated
If you found DirectoriesBot in your access log, this page explains what it was doing there. It is the crawler Driven Success LLC runs to check reciprocal links, and it only ever visits sites that a directory operator has listed.
What the bot does
Directories runs link directories in which a listing is granted on the understanding that the listed business links back. DirectoriesBot is the part that checks whether that link is still there.
It fetches one page, reads the links on it, and records whether one of them points at the listing. That is the whole job. It does not index your site, it does not train anything on it, and it does not store your page.
We keep the outcome, the URL we fetched, the response code, and the single link that matched if one did. No personal data is involved, because a link is not personal data.
How to recognise it
Every request carries the same User-Agent, and the address inside it is this page.
The last field names the directory whose listing is being checked, so you can see which of your business relationships prompted the visit without having to ask us.
- DirectoriesBot/1.0 (+https://directories.prosvita.io/bot; verification for your-directory.example) — The exact header sent, with the directory's own domain in the final field.
What it fetches
Only the exact URL a directory operator entered for a listing. We do not follow links out of that page, we do not walk your sitemap, and we do not guess at other addresses on your domain. One listing is one page.
We follow up to 3 redirects, so a page that has moved is still found. We send no cookies and store none.
If you have several listings across different directories, each is checked separately, but never at the same time — see below.
How it treats robots.txt
We read robots.txt before every page and obey it. A group addressed to DirectoriesBot wins over the wildcard group, Allow and Disallow are matched longest-first, and an empty Disallow means everything is allowed.
A robots.txt that is missing or returns a 404 means no restrictions, which is what the standard says it means. A robots.txt that errors or times out means we do not fetch the page at all — we would rather record nothing than guess.
We cache the file for 24 hours, so a change takes at most that long to reach us. We read Crawl-delay but do not need it, because one connection per host at a time is already stricter than any value you could set.
How often it visits
We open one connection to a host at a time and never more. A check that arrives while another is already running is postponed rather than run alongside it, so we cannot put more than one request a time on your server no matter how many of your pages are listed.
Each page is checked on the operator's schedule — monthly, weekly or daily depending on their plan — and an operator can ask for a single re-check by hand. There is no continuous crawl.
Each request is bounded, so a slow or enormous response costs you nothing:
- 15s — The whole fetch, redirects included, is abandoned after this.
- 3 — The most redirects we will follow before giving up.
- 5MB — We stop reading a response after this much and close the connection.
Opting out
The quickest way to stop us is robots.txt, and it takes effect within 24 hours. Disallowing DirectoriesBot from a path means we stop fetching it, and the operator who listed you sees the listing as unverified rather than as a broken link.
If you would rather not be listed at all, that is between you and the directory that listed you — we host the directory, we do not choose what goes in it. Write to us anyway and we will pass it on, and we will block your domain from being fetched in the meantime.