Our crawler
pubsleuth reads public files and pages to see which ad networks and affiliate programs a site uses. If you saw our requests in your logs, this page explains them.
What we fetch
/ads.txtand/app-ads.txt, which exist to be read by the ad industry.- The
sellers.jsonfiles ad networks publish. - Your home page, your sitemap, and a few public pages it lists, such as reviews or comparisons.
- The redirect of a short or affiliate link on those pages, one request per hop, to see where it leads.
What we don't do
- We never log in, submit forms, click ads or buy anything.
- We never try to get around a bot challenge or a block. A blocked site is recorded as blocked.
- We don't store copies of your pages for publication; we keep the evidence we found (for example an ads.txt line or a link) and the page it was on.
How often
A typical site gets a handful of requests when it is first checked and a few more when it is checked again, days or weeks later. Requests to any one host are spaced out, never in parallel bursts.
Opting out
To have your site removed from pubsleuth, use the removal form. Once we have checked the request, the site disappears from every service. You can also email [email protected] about anything our crawler did.