Have you optimized your WordPress Robots.txt file for SEO?
If you haven’t, you are ignoring an important aspect of SEO. Robots.txt file plays a significant role on your site’s SEO.
You are lucky that WordPress automatically creates a Robots.txt file for you. Having this file is half of the battle. You have to make sure that Robots.txt file is optimized to get the full benefits.
Robots.txt file tells search engine bots what pages to crawl and what pages to avoid. In this post, I will show you how to edit and optimize Robots.txt file in WordPress.
Robots.txt file is a text file which instructs search engine bots how to crawl and index a site. Whenever any search engine bots come to your site, it reads the robots.txt file and follow the instructions. By using this file, you can specify bots which part of your site to crawl and which part to avoid. However, the absence of robots.txt will not stop search engine bots to crawl and index your site.
I’ve already said that every WordPress site has a default robots.txt file in root directory. You can check your robots.txt by going to http://yourdomain.com/robots.txt. For example, you can check our robots.txt file from here: http://roadtoblogging.com/robots.txt
If you don’t have a robots.txt file, you’ll have to create one. It’s very easy to do. Just create a text file in your computer and save it as robots.txt and upload it to your root directory. You can upload it via FTP Manager or cPanel File Manager.
Now let’s see how to edit your robots.txt file.
You can edit your robots.txt file by using FTP Manager or cPanel File Manager. But it’s time-consuming and a bit difficult.
The best way to edit Robots.txt file is, using a plugin. There are several WordPress robots.txt plugins out there. I prefer Yoast SEO. This is the best SEO plugin for WordPress. I’ve already shared how to set up Yoast SEO.
Yoast SEO allows you to modify the robots.txt file from your WordPress admin area. However, if you don’t want to use Yoast plugin, you can use other plugins like WP Robots Txt.
Once you’ve installed and activated Yoast SEO plugin, go to WordPress Admin Panel > SEO > Tools.
Then you need to click on “Create robots.txt file”.
Then you will get the Robots.txt file editor. You can configure your robots.txt file from here.
Before editing the file, you need to understand the commands of the file. There are three commands mainly.
This robots.txt file instructs all search engine bots to crawl the site. In the 2nd line, it tells search engine bots not to crawl the /wp-admin/ part. In 3rd line, it instructs search engine bots to crawl and index whole website.
A simple misconfigure in Robots.txt file can completely deindex your site from search engines. For example, if you use the command “Disallow: /” in Robots.txt file, your site will be deindexed from search engines. So you need to be careful while configuring.
Another important thing is optimization of Robots.txt file for SEO. Before going to the best practices of Robots.txt SEO, I’d like to warn you about some bad practices.
Now let’s see how you can make Robots.txt file SEO friendly.
Here’s an example of ideal Robots.txt file for WordPress.
You can check RTB Robots.txt file here: http://roadtoblogging.com/robots.txt
After updating your Robots.txt file, you have to test the Robots.txt file to check if any content is impacted by update.
You can use Google Search Console to check if there is any “Error” or “Warning” for your Robots.txt file. Just login to Google Search Console and select the site. Then go to Crawl > robots.txt Tester and click on “Submit” button.
A box will be popped up. Just click on “Submit” button.
Then reload the page and check if the file is updated. It might take some time to update the Robots.txt file.
If it hasn’t updated yet, you can enter your Robots.txt file code into the box to check if there are any errors or warnings. It will show the errors and warnings there.
If you notice any errors or warnings in the robots.txt file, you have to fix it by editing the robots.txt file.
I hope this post helped you to optimize your WordPress robots.txt file. If you have any confusion regarding, feel free to ask us via comments.
However, if you want to make your WordPress blog SEO friendly, you can read our post on How to Setup WordPress Yoast SEO Plugin.
If you find this post helpful, please help me by sharing this post on Facebook, Twitter or Google+.