Categories
SEO

Understanding the Basic Differences Between Meta Robots Tag and X-Robots-Tag HTTP Header

Many SEOs will likely know about the meta robots tag, and fewer would have heard of the X-Robots-Tag. And while fundamentally they both can help achieve the same results, this post will discuss what the basic differences between these two are.

Difference 1: Implementation

What I mean by that is this:

  1. Meta robots tag is an HTML code — which can be coded on your sites’ pages. Usually, it’s quite plausible for you to get access to this capability. As a Matter of fact, if you have WordPress, and you’re using Yoast SEO, you can get the option to dictate meta robots tags’ directives using the “advanced” feature.
  1. X-Robots-Tag is an HTTP Header. This is not something that you can code it in. It needs to be configured. It’s also not something you can typically get access to. It’s best suited for experienced developers.

Difference 2: Checking the Presence of These Tags on a Web Page

The meta robots tag can be found by simply going to the source code. The X-Robots-Tag cannot be found there. You’d have to check for HTTP Response Headers in your favorite browser. As an example, see the screenshot below for the https://feedthecuriosity.com/sitemap_index.xml URL.

x-robots-tag response header
Going to the “network tab” reveals the use of x-robots-tag.

Difference 3: Use Cases

Meta robots tag while handy, and easy to implement, cannot be placed on pages that are not HTML, such as .xml extension (as evident from the screenshot above, or PDFs).

Conversely, as you might have guessed, X-Robots-Tag comes in handy for non-HTML pages. This difference right here is probably the biggest one that separates the two.

Difference 4: Scalability

While meta robots tag can also be applied to multiple pages at once using logic in the URLs, it may end up using more resources. X-Robots-Tag is more convenient in that manner, and can easily incorporate for complex regex rules.

Difference 5: Search Engines

It looks like not all search engines support the X-robots-Tag. Google definitely supports both, but if you do care about others as well, you’ll have to reconsider which type of robots command you’re using to control crawling & indexing.

not all search engines support x-robots-tag
Source of the screenshot: https://ahrefs.com/blog/meta-robots/

You May Also Want to Check Out:

What Are Some of the Common Meta Robots Directives Used?

There are a handful of these, but let’s cover the ones that are more commonly used. (Moz does an incredible job of going over a lot more).

  1. noindex: Tells search engines to not to index the page.
  2. index: Tells the search engines to index the page.
  3. nofollow: Primarily used for dictating to crawlers to not to follow any links on a specific page.
  4. follow: instructs search engines to follow links (even if the prior instructions are to not to index the page).
  5. noarchive: Instructs the search engines to not to display a cached copy of a page in the search results.

It should be noted that it’s possible to use more than 1 directive together. As an example, see the screenshot below from the source code of https://feedthecuriosity.com/.

meta robots example from the homepage
This pretty much tells the search engines to index and follow.

A few other things to take note here are:

  1. By default, it is assumed that the robots directive is to index and follow. In other words, if a meta robots tag command is not present, it would be assumed that the page is open for indexation, and that all links can be followed.
  2. The robots tag is always implemented in the head section.

What About X-Robots-Tag?

It can support all of the same directives as a meta robots tag. But as mentioned, it comes in handy for non HTML pages, and to scale things by leveraging regular expressions.

Conclusion

I hope this post helps you capture the basic differences between the two. Personally, I haven’t had the pleasure to execute the X-Robots-Tag, but it’s something I wouldn’t mind getting the first-hand experience of.

Regardless of the situation, there are two types of robots directives available. If one doesn’t work, try the other. And if someone asks you to noindex a non-HTML page, you’ll know where to look.