A Brief Guide to Ahrefs Site Audit’s “Discoverable URLs by Crawl Source”

The Ahrefs site audit is a marvelous wonder for any SEO with tons of visualizations, explanations, and illustrations. The power is not only in the aesthetics, but also in the thoroughness, technicality, customizations, meticulous attention to even tiniest of details, as well as, in overall appeal.

It isn’t easy to put all the awesomeness into words for the purposes and intent of this post, but if you ever get a chance to tinker with it, I highly recommend it!

Anyhow, coming back to the primer of this post, from the myriad of data points and reports and issues available, what uniquely stood out to me was its “Discoverable URLs by crawl source.”

What Exactly Is “Discoverable URLs by Crawl Source” in Ahrefs?

As the name suggests, this report identifies how many internal URLs were discovered during a crawl — by source type. A source could be XML sitemaps, your website, backlinks, etc. As things stand today, Ahrefs offers 5 source types — that a project owner can choose from their project settings.

Ahrefs site audit URL sources
As shown, Ahrefs offers 5 source types for your site’s crawl.

The tool will crawl your website from all selected URL sources. Once the crawl is complete, eventually, as it pertains to the central point of this post, Ahrefs will show you how many internal URLs were found by each source.

This data is located under Reports > Internal Pages and will look similar to the screenshot below.

how the Ahrefs discoverable urls by crawl source report looks like
How the Ahrefs discoverable URLs by crawl source report looks like.

A Few Definitions You’d Need To Know

When you’re playing around with the report/horizontal bar chart, you’ll notice that the data could be broken down by 3 types of URLs — although, not all may apply to you. They are:

  1. Found: this will provide you with the count of URLs found via each source.
  2. Only: this will give you a list of URLs that were found only via a specific source, and not others.
  3. Missing: this will furnish a list of URLs discovered in other sources, but were missing in the specific one you’re currently looking at.

The bar chart is color-coded accordingly for each URL source, and the data can be visualized either as a percentage or absolute values. Regardless of which option you elect to go with though, hovering your mouse over the bar graph will invoke the exact number of URLs for either found, only, and or missing (depending on what applies to you). See the image below for reference.

how the data is displayed in the Ahrefs discoverable URLs by crawl source
How the data is broken down in the Ahrefs discoverable URLs by crawl source.

The Significance of the Ahrefs Site Audit’s “Discoverable URLs by Crawl Source”

Coming to the principal theme here: Why am I even talking about this?

The entire idea of this reporting is to impart insights into how easily are your site’s internal URLs discoverable by the Ahrefs’ crawler by each source. The ideal state is that all URLs should be found with each source and not skewed towards one or the others.

As a very simplistic example: Say you have a total of 50k URLs on your site. In a perfect world, all 50K URLs should be crawled by each source type, that is, in XML sitemaps, the primary project URL, backlinks, etc. I wouldn’t hold much weight on the backlinks, but definitely, the site and the XML sitemaps.

IF your results are skewing towards missing or only by a large margin towards a particular source, then according to Ahrefs, you might have issues with proper internal linking, which could propagate in the form of having orphaned pages (pages that have zero links from your own site, so basically are on their own, flying solo), or indexation issues.

As a real-world instance, this website received the same found URLs from the site itself and XML sitemaps. Of course, it is a very small website, so easier said than done. However, for a big website that is serious about crawlability, using this feature report in Ahrefs can be an indispensable asset to realize which missing pieces need to be completed to enhance URL discoverability in the search further.

You May Also Want to Check Out:


As I have advocated before, like many SEOs, I find that the Ahrefs SEO tool is paramount to my SEO success and strategy. In fact, lately, they have been rolling out a bunch of new features to help site owners even more in their organic growth ambitions.

Speaking of, to increase organic visibility, URL discoverability is critical to any size website. And one of the best ways to find out how you’re faring in that aspect? Try the Ahrefs site Audit! Pick out your URL sources, run a crawl, and once finished, head over to Reports > Internal Pages to see the findings on “Discoverable URLs by crawl source.”