A robots.txt file is always supposed to live at the root of your site. So for example, the robots.txt for Feed The Curiosity can be found at https://feedthecuriosity.com/robots.txt. This file plays a special role in instructing search engine bots on what to crawl and what not to.
A good rule of thumb is to remember the following:
- This file is nothing but a text file.
- It always needs to be present at the root of your website, as stated.
When it comes to WordPress, there are two methods you can use to edit your robots.txt.
- Method 1: Using the Yoast Plugin (yes, even the free version lets you make edits).
- Method 2: Gaining back-end access to your site, which technically, is the most fundamental method for any website, WordPress or not. Methodologically speaking, you’d need access to upload the updated robots.txt file, not to make the actual edits.
Let’s dive into both, to understand what the process looks like.
Method 1: Using Yoast SEO Plugin to Edit the robots.txt File
When the plugin is installed, inside your WP dashboard, you’d typically get a separate line item for “SEO.” Click on it, to get more options.
Next, select tools and then eventually File editor, to open your robots.txt.
The robots.txt will appear as a text field area, where you can make the necessary adjustments you’d want, and then, select the “Save changes to robots.txt” button.
That’s all you need to do as far as leveraging the Yoast Plugin goes. However, there would be a few additional tasks you should perform, to ensure search engines can read the updated robots.txt. They are (in order of):
- Clear Your WordPress Cache.
- If you have access to webmaster tools, namely GSC, and Bing Webmasters Tools, make them fetch the latest robots.txt by logging in.
Method 2: Using Back-End Access/FTP Access to Upload Updated robots.txt File
If you do not have access to Yoast for some reason, or, if you do, and you cannot access the File editor, there is an alternative available.
To create and or to update the existing file, you can use any text editor of your choice like Notepad++, and then save the file as—>robots.txt. This is actually quite important. You have to save your file name verbatim, as mentioned. Also, the file’s encoding needs to be in UTF-8, but don’t worry, most of the text editors can support that.
In any case, once you’re happy with your changes, save the file as recommended, and upload it at the root of your site using an FTP/File manager service. In a lot of the cases, your domain registrar (for example GoDaddy), may even have an FTP like service. It doesn’t matter how you gain access to your back-end, more than the fact that you can.
After you’re connected to your back-end, upload the file at the root, save changes, and you should be all set. If you do need a tutorial on using a separate FTP software, you can check out my post on using FileZilla for WordPress.
And again, with this method too, remember to flush you WordPress cache (as a best practice), and have search engines fetch the latest robots.txt.
A Few FAQs regarding robots.txt
Both Bing, and Google have a robots.txt tester. I personally prefer Google’s. In any event, go to the tester, and manually plugin some URLs to check if:
1. Are the URLs you expect to be blocked, are indeed getting blocked/disallowed.
2. Are the URLs you expect to be crawled, can, in fact, be crawled.
Further, as it happens, you can plug and play with your code here first, to test if you aren’t making any technical mistakes, before, you update and submit a modified robots.txt file.
To view a live version for any website, type in website.com/robots.txt, to see what their current configurations are.
Yes. The general recommendation is to include your XML Sitemap URLs inside your robots.txt. And while technically, there is an option to submit them individually from search engine specific webmaster tools, having them inside robots.txt can serve as a backup for Google, and Bing, especially, if something goes wrong within webmaster tools, or you lose access to it, etc.
Is it mandatory to include the XML sitemaps? No.
Yes. To manage crawling behavior for subdomains, you need a separate robots.txt. So for instance, if you have a blog on a subdomain, and you want to more precisely dictate how search engines can crawl its URLs, your robots.txt should reside at blog.yourwebsite.com/robots.txt
You May Also Want to Check Out:
Conclusion
Editing robots.txt file and submitting changes for the same, in a WordPress site, isn’t hard at all. Chances are, you’re likely using the Yoast SEO plugin, so might as well take advantage of the tool to also update your robots.txt.
Outside of the plugins in general, the other option is to use a text editor, save changes, and upload the file to the root directory via FTP or a file manager that gives you back-end access.