Our data

What pubsleuth holds, how much of it there is, where it comes from, how complete and how fresh it is, and what we are doing to make it better. Every figure below is counted from the live data, not typed in by hand, and the page says on which day it was counted.

Counted on , from the latest daily load

The size of it, at a glance

Three services read three kinds of public evidence. These are the headline counts of what each has found and kept, using the same definitions the products use: an AdSense site is one where the AdSense code and ID are really there, a network site is one found on a network (and verified when the site itself confirms it), and an affiliate site is one confirmed to earn from a program.

  • 819,808AdSense sitesWith the AdSense code and a publisher ID found
  • 596,812Publisher IDs seenAccounts that appear on at least one site
  • 370,647Sites with a siblingOn an ID that runs two or more sites
  • 220,549Network sites105,259 verified, across 173 ad networks
  • 400,813Tool pages foundOn 10,875 verified sites
  • 355,056Confirmed affiliate sitesPlus 8,029 likely and 115,323 candidates
  • 89Affiliate programsAmazon Associates, Awin, CJ and more
  • 18,841Affiliate site groupsSites that share an affiliate ID

What each service holds about a site

A record is always a website, found by its domain. What we keep on it depends on the service, and a site can be in all three at once: an AdSense site, on Mediavine, and an Amazon affiliate.

AdSense Index

819,808

sites

Every site where the AdSense code and a publisher ID are really on the page.

  • The site’s title, niche, sub-niche, cluster of similar sites, type and language
  • Its monthly visits estimate and, for some, a three-month trend
  • Audience countries and each country’s share
  • An estimated revenue range and page RPM
  • Whether the ads are really being served
  • Every publisher ID found and every other site on it

Network Scout

220,549

sites on 173 ad networks, 105,259 of them verified

Which ad networks a site uses, found in each network’s sellers.json and verified by the site’s own ads.txt or page code.

  • The networks a site is verified on, and the ones it has left
  • The ads.txt systems it names
  • Tool pages: calculators, converters and the like
  • Estimated traffic and revenue, and a niche where we have labelled one
  • Sites that share an analytics or tag-manager ID

Affiliate Scout

355,056

confirmed sites across 89 programs

Which affiliate programs a site earns from, found from the links on its own pages.

  • The programs and merchants its links lead to
  • The affiliate IDs and tags found, and the other sites that carry them
  • A verdict: confirmed, likely, candidate or mirror
  • Products it links to, niche, and a rough revenue band

What is never in any of them: an email address, a phone number or any other way to contact the person behind a site. pubsleuth holds facts about websites that the websites themselves publish. It is not a list of people.

How it has grown

Each chart is a running total of the sites we have found, drawn by the day we first found each one. They show our coverage growing, which is not the same thing as the web growing.

AdSense sites819,808 AdSense sites found so far, from the first on 3 October to 20 November.
3 October819,808 on 20 November
Verified network sites105,259 verified network sites found so far, from the first on 24 September to 20 November.
24 September105,259 on 20 November
Confirmed affiliate sites355,056 confirmed affiliate sites found so far, from the first on 21 October to 19 November.
21 October355,056 on 19 November

The index is young. The first AdSense site went in on 3 October, and nearly all of what you see here arrived in the weeks since. Growth was steepest at the start, when a crawler can work through the easiest sources first: public rankings of popular sites, the ads.txt files of sites already known, and Google’s own list of the accounts it pays. It slows as the easy sources run out, and the sites that remain are harder to find. A flattening line is the sign of a crawler running out of easy answers, and it does not mean the web has stopped producing sites.

Two things follow, and both matter when you read a figure on pubsleuth. First, the counts are what we have found, and they are a lower bound on what exists. We say “the AdSense sites we track” and mean it. Second, because most of the index is only weeks old, a count that rises from one day to the next mostly reflects coverage and not change on the web. As the index matures, the daily changes will start to mean more: sites that joined a network, left one, or switched their ads off.

How complete it is, fact by fact

Finding a site is the first half. The second is knowing things about it. This is the share of the 819,808 AdSense sites that has each fact today, with the count behind each bar. We publish the gaps as readily as the strengths, because you should know which columns you can lean on.

  • A niche99.5% 816,168
  • A sub-niche99.5% 816,164
  • A site type99.5% 816,168
  • A language99.5% 816,168
  • At least one publisher ID99.6% 816,667
  • A monthly visits estimate99.4% 815,621
  • Audience countries49% 399,265
  • An estimated revenue range99.4% 815,621
  • A serving-ads verdict99.9% 819,650
  • A three-month trend2% 19,911

The facts that describe what a site is are the most complete: 99.5% of sites have a niche, 99.5% a sub-niche, 99.5% a type and 99.5% a language, and 99.6% have a publisher ID, which is how they were found. Traffic and revenue come next: 99.4% of sites have a monthly visits estimate and 99.4% a revenue range, because the model needs only a site’s public ranking to produce them. The serving-ads verdict, which comes from Google’s own sellers.json, covers 99.9%.

Two facts are thinner, and we would rather say so. Audience countries cover 49% of sites, and a three-month trend covers 2%. A trend needs a measured history, and only a minority of sites have one. If a filter or a sort on one of these facts returns fewer sites than you expected, this is why. The sites without a niche are waiting for a label.

Traffic deserves a sentence of its own. For most sites the visits figure is our own model, built from public rankings, and the pages mark it with a tilde to say so. Where a measured figure exists for a site we use it. Either way it is an estimate of what a site gets, and the revenue range built on it is an estimate of an estimate. That is why revenue is always a range, and why we keep telling you to compare sites with each other and not to quote one site’s figure as fact.

What happens to the sites we cannot verify

Network Scout started from 220,549 sites that a network’s files or our own discovery pointed at, and checked each one. 105,259 of them, 48%, passed the test for being really on a network. The rest are not hidden: each is sorted by what we found.

What we foundSitesShareWhat it means
Verified105,25948%The site’s own ads.txt carries the network’s direct line, or its pages load the network’s tag. The site and the network confirm each other.
Domain gone32,45315%The domain no longer answers: it has expired, been parked or been taken down.
No ads.txt24,05111%The site has no ads.txt file we could read, so there is nothing to check the network’s claim against.
Could not reach it23,58911%The site did not answer when we asked, or answered with an error. It may be tried again.
Listed, not verified20,2589%The network’s own files name the site, but the site’s ads.txt does not carry the network’s line. A listing alone is a weaker claim, so we show it as listed.
Refused us14,9397%The site’s server turned away our request for its ads.txt, for example with a bot wall or a rate limit, so there was nothing to check. We don’t try to get around it.

The headline count of network sites is every site found on a network, as the dashboard counts them; the figures in this section count verified sites only. A site’s record and the Network filter show every network that names the site now, verified or only listed, and the record’s evidence table says which source named each one. A site that was verified and later stopped carrying a network’s line is not deleted. Who joined and who left a network is shown only once a change is confirmed: the network’s own sellers.json and the site’s ads.txt agree, or one of them shows the same change on two reads at least 3 days apart.

One gap to be plain about: of the verified sites, 92,684 (88%) have a niche label so far. The rest are verified and real, and show as “niche pending” until they are labelled.

How sure we are that a site earns from affiliate programs

Affiliate Scout does not say a site earns from a program because a page mentions it. It grades the evidence, and the grade is on every record.

VerdictSitesWhat we saw
Confirmed355,056At least one proven affiliate link, such as an Amazon link carrying a real tag or a network link with a publisher ID.
Likely8,029Two strong signals that agree, such as a disclosure and sponsored links, but no single proven link.
Candidate115,323Commercial markers only, or a single strong signal. Worth another look; not counted as an affiliate site.
Mirror1,114A copy of another site we already hold, kept so the copy is not counted twice.

Only confirmed sites are counted as affiliate sites in the headline figures and in the products. Likely and candidate sites are kept because a later check may confirm them: the crawler checks a site again once its last check is more than 30 days old. Links in comments never count, and short or cloaked links are followed to where they lead before they are judged.

How fresh it is

The crawlers keep running, Affiliate Scout's in rounds that start a minute after the last one ends, and what they find reaches the website once a day in a single load: a snapshot that replaces the last one only when it has been checked as a whole. The date at the top of this page is the date of that load. Home page and dashboard counts are dated the same way.

Freshness of an individual site is a different question. Every AdSense site records when we first found it and when we last checked it. Today 276,318 of 819,808 AdSense sites (34%) were first found in the last 30 days, and 276,320 (34%) were last checked in that time. That is not a boast about re-checking: it is the index’s age showing. It is only weeks old, so nearly every site’s last check is recent. As it ages, going back to sites we have not looked at for a while becomes the main job, and the check date on each record is how you can tell how stale a result might be. Over the last week 7,453 sites were added, which is the pace of discovery now that the first rush is over.

How we collect it

Everything starts from files and pages that websites publish on purpose, for the advertising industry or for any visitor. We do not log in anywhere, buy lists of people or read anything that is not public.

Finding AdSense sites

An AdSense site announces itself in three places. The code on its pages carries the AdSense script and the publisher ID. Its ads.txt file, at the root of the domain, lists the companies allowed to sell its ad space, and the line for Google carries the ID. And Google publishes a sellers.json file naming the accounts it pays. We read all three. To know which sites to look at, we start from public rankings of popular sites, from the ads.txt files of sites we already know, and from the files that ad networks publish. Where a site has no AdSense code at all, we do not guess: it is not an AdSense site.

Finding ad network sites

Each ad network that publishes a sellers.json file lists the sites it pays. We read those files, then go to the sites themselves and read their ads.txt and home pages to see whether the network’s line and tag are there. We also read the pages of verified sites to find tool pages, and we keep the history of which networks a site has been on, so a move from one network to another is a recorded event and not a mystery.

Finding affiliate sites

Affiliate Scout reads a site’s home page and the commercial pages its sitemap points to, such as reviews and comparisons. It follows short and cloaked links to their destination, looks for the tags and IDs affiliate programs use, and records the evidence for each finding: the page, the link and the program.

How we behave when we visit

Every request carries the User-Agent name PubSleuthBot, so you can see us in your logs. A site that does not want the ad-network and affiliate checks to read its pages can block that name in its robots.txt, and they stop reading them. Network Scout still reads the site’s ads.txt and app-ads.txt files. The AdSense checks don’t read robots.txt. They read a site’s ads.txt, its home page and up to two linked pages, and, to check that a site really shows AdSense ads, open the home page and at most one article in a browser that runs the page’s own scripts and loads no images. That browser stops every ad request before it reaches Google, so no ad is shown or counted. If you run a site and have a question about our crawler, the contact page reaches us.

How we check it

A crawler finds things. Most of what makes data trustworthy is what the system refuses to believe. These are the rules that cost us results and prevent wrong ones.

  • An ads.txt line alone proves nothing. Ad networks list their own AdSense accounts in the ads.txt files of every site they serve. If we counted those, one network’s account would seem to own thousands of unrelated sites. An ID is tied to a site only when the site’s own page code carries it, a public browser crawl saw the page ask for ads with it, or Google’s sellers.json names the account for that site.
  • Carrying the code is not serving ads. We check the ID against Google’s sellers.json. A site whose ID is not there almost never gets an ad, so its estimated revenue is zero, and the record says “not serving”.
  • A network needs two sides to agree. A site is verified on a network when its own ads.txt carries the network’s direct line or its pages load the network’s tag. A network’s own list of sites, with no confirmation from the site, is shown as “listed”.
  • Affiliate evidence is graded. One proven link is confirmed. Two agreeing signals are likely. One strong signal, or commercial markers alone, make a candidate, which is not counted.
  • Shared IDs are clues, not proof. An agency can put one analytics ID on every client’s site. IDs shared by very many sites are ignored as templates, and every group shows the IDs that joined it, so you can judge the link yourself.
  • Every estimate is labelled. Traffic from our model carries a tilde. Revenue is a range. Affiliate revenue is a rough band meant only to rank sites against each other.
  • Removals are complete. When a domain is removed after a site owner’s objection, it disappears from every service within about a minute.

How we are improving it

This is where the numbers above are going, and where we think they are weakest. None of what follows is a promise of a date. It is the work in progress, listed so you can judge for yourself whether we are working on what matters to you.

What has changed since we began

  • The AdSense index went from its first site on 3 October to 819,808 sites. Daily loads now replace the whole picture once a day.
  • Network Scout grew to 105,259 verified sites across 173 networks, and keeps the history of the networks each has left.
  • Affiliate Scout reached 355,056 confirmed sites across 89 programs, with the evidence for each, and now links sites that share an affiliate ID into 18,841 groups.
  • Each week we hand the AdSense index to Network Scout: sites whose ads.txt names a network we track, and that Network Scout has not yet looked at, are queued for checking.

What we are working on

  • Labels for the sites that have none. Every site without a niche, sub-niche or type is waiting for one. This is the largest gap in how much you can filter by.
  • Traffic and audience countries for more sites. The audience country and three-month trend filters work only where we hold the data. Widening that coverage makes every one of them more useful.
  • More of each site, not just more sites. Wider keyword and page coverage in Network Scout: pages that are not transactional, and more pages for each site, so the tool pages and topics on a site are a fuller picture.
  • Going back. As the index ages, checking sites again becomes the main work: to see which have switched ads off, moved networks or gone.
  • Finding sites we have not found. Discovery has slowed from its first weeks. New sources of sites are how it speeds up again, and more networks and affiliate programs are added as we meet them.

If you find a mistake, such as a site filed under the wrong niche, an ID that is not really on the page or a figure that looks wrong, tell us through the contact page. It is the fastest way we have of finding the rules that need changing.

What we do not have

Better to say it here than have you discover it later.

  • Contact details. No emails, phone numbers or addresses. We do not hold them and will not.
  • Every AdSense site on the web. We hold the ones we have found. Sites that carry the code behind a login, or that no source points us to, are not here.
  • Measured revenue. No one outside a publisher can see what the publisher earns. Our revenue is a range, built from traffic, pages per visit and the niche’s typical page RPM.
  • Domain Authority. Or any other third-party SEO score. We show traffic, rank, trend, audience and structure instead.
  • Keyword search volumes. We show which keywords have supported earning sites, not how often people search for them.
  • Anything private. If a page needs a login to be read, we do not read it.