Everything starts from files and pages that websites publish on purpose, for the advertising industry or for any visitor. We do not log in anywhere, buy lists of people or read anything that is not public.
Finding AdSense sites
An AdSense site announces itself in three places. The code on its pages carries the AdSense script and the publisher ID. Its ads.txt file, at the root of the domain, lists the companies allowed to sell its ad space, and the line for Google carries the ID. And Google publishes a sellers.json file naming the accounts it pays. We read all three. To know which sites to look at, we start from public rankings of popular sites, from the ads.txt files of sites we already know, and from the files that ad networks publish. Where a site has no AdSense code at all, we do not guess: it is not an AdSense site.
Finding ad network sites
Each ad network that publishes a sellers.json file lists the sites it pays. We read those files, then go to the sites themselves and read their ads.txt and home pages to see whether the network’s line and tag are there. We also read the pages of verified sites to find tool pages, and we keep the history of which networks a site has been on, so a move from one network to another is a recorded event and not a mystery.
Finding affiliate sites
Affiliate Scout reads a site’s home page and the commercial pages its sitemap points to, such as reviews and comparisons. It follows short and cloaked links to their destination, looks for the tags and IDs affiliate programs use, and records the evidence for each finding: the page, the link and the program.
How we behave when we visit
Every request carries the User-Agent name PubSleuthBot, so you can see us in your logs. A site that does not want the ad-network and affiliate checks to read its pages can block that name in its robots.txt, and they stop reading them. Network Scout still reads the site’s ads.txt and app-ads.txt files. The AdSense checks don’t read robots.txt. They read a site’s ads.txt, its home page and up to two linked pages, and, to check that a site really shows AdSense ads, open the home page and at most one article in a browser that runs the page’s own scripts and loads no images. That browser stops every ad request before it reaches Google, so no ad is shown or counted. If you run a site and have a question about our crawler, the contact page reaches us.