About DirectoriesBot
Last updated
If you found DirectoriesBot in your access log, this page explains what it was doing there. It is the crawler Driven Success LLC runs to check reciprocal links, and it only ever visits sites that a directory operator has listed.
What the bot does
Directories runs link directories in which a listing is granted on the understanding that the listed business links back. DirectoriesBot is the part that checks whether that link is still there.
It fetches one page, reads the links on it, and records whether one of them points at the listing. That is the whole job. It does not index your site, it does not train anything on it, and it does not store your page.
We keep the outcome, the URL we fetched, the response code, and the single link that matched if one did. No personal data is involved, because a link is not personal data.
How to recognise it
Every request carries the same User-Agent, and the address inside it is this page.
The last field names the directory whose listing is being checked, so you can see which of your business relationships prompted the visit without having to ask us.
- DirectoriesBot/1.0 (+https://directories.prosvita.io/bot; verification for your-directory.example) — The exact header sent, with the directory's own domain in the final field.
How to verify it
A User-Agent is a claim anybody can type. Every request we send also carries a cryptographic signature, so you can confirm it is really us — without contacting us, and without trusting a list of addresses that changes.
Each request carries a Signature header, a Signature-Input header naming exactly what was signed, and a Signature-Agent header pointing at where our keys live:
- "https://directories.prosvita.io" — The address our Signature-Agent header names, quoted exactly as it is sent.
- https://directories.prosvita.io/.well-known/http-message-signatures-directory — Our public keys, as a JSON Web Key Set that is itself signed with every key it lists.
Fetch that directory, take the key whose kid matches the keyid in Signature-Input, and check the signature over the request. These are the keys we sign with today:
- LxO8V1TZkoV3edjw4KHieLi-gBtmDhu3wGwmcOnry_w — A key thumbprint, as it appears in the keyid parameter of Signature-Input.
If your site is behind Cloudflare or another service that checks these signatures, this happens for you and there is nothing to set up. We follow Web Bot Auth, the bot profile of RFC 9421 HTTP message signatures.
What it fetches
Only the exact URL a directory operator entered for a listing. We do not follow links out of that page, we do not walk your sitemap, and we do not guess at other addresses on your domain. One listing is one page.
We follow up to 3 redirects, so a page that has moved is still found. We send no cookies and store none.
If you have several listings across different directories, each is checked separately, but never at the same time — see below.
How it treats robots.txt
We read robots.txt before every page and obey it. A group addressed to DirectoriesBot wins over the wildcard group, Allow and Disallow are matched longest-first, and an empty Disallow means everything is allowed.
A robots.txt that is missing or returns a 404 means no restrictions, which is what the standard says it means. A robots.txt that errors or times out means we do not fetch the page at all — we would rather record nothing than guess.
We cache the file for 24 hours, so a change takes at most that long to reach us. We read Crawl-delay but do not need it, because one connection per host at a time is already stricter than any value you could set.
How often it visits
We open one connection to a host at a time and never more. A check that arrives while another is already running is postponed rather than run alongside it, so we cannot put more than one request a time on your server no matter how many of your pages are listed.
Each page is checked on the operator's schedule — monthly, weekly or daily depending on their plan — and an operator can ask for a single re-check by hand. There is no continuous crawl.
Each request is bounded, so a slow or enormous response costs you nothing:
- 15s — The whole fetch, redirects included, is abandoned after this.
- 3 — The most redirects we will follow before giving up.
- 5MB — We stop reading a response after this much and close the connection.
Opting out
The quickest way to stop us is robots.txt, and it takes effect within 24 hours. Disallowing DirectoriesBot from a path means we stop fetching it, and the operator who listed you sees the listing as unverified rather than as a broken link.
If you would rather not be listed at all, that is between you and the directory that listed you — we host the directory, we do not choose what goes in it. Write to us anyway and we will pass it on, and we will block your domain from being fetched in the meantime.