Custom Robots.txt Generator For Blogger (BlogSpot) Blog

Blogger Robots.txt Generator Tool

Works with both, .blogspot.com and custom domains.

Enter your Blogger blog’s address (URL) below. [Including https:// and www. (if custom domain)]

Copy

What Is Robots.txt And How Is It Useful For Blogger (BlogSpot)?

Robots.txt is a text file which helps the bots/crawlers of the search engines such as Google and Bing to crawl and index your site effectively.

It is a set of instructions or rules, which the crawlers and search engine bots visiting your site, follow.

The Robots.txt file is also used by crawlers from other types of websites such as social media sites like Facebook, Twitter and SEO sites like online keyword research tools.

The main function of the Robots.txt file is to guide the crawlers and the bots, about which portion of your site should be available on the search engines and which parts should not show up on the search.

For example, a blogger would want all her/his articles to show up on the search engine, but would definitely not want other pages like the blog’s internal search pages to show up on Google!

Let’s consider a scenario:

Suppose a person visits your website and uses the search bar on your site to look up a random word unrelated to the niche of your website.

He/she might end up with something like this:

And in Blogger, the URL of that page will be: https://www.example.com/search?q=hallelujah

What will happen next is, the crawler in the background will most probably crawl this search results page as well and put it in its index. And due to that, these low-quality pages that will eventually get stored in Google’s index will affect the authority of the blog.

Here’s an example:

But with Robots.txt file present, it will instruct the web crawlers not to put such pages in their index and thus, preventing the indexing of such low-quality pages.

Thus, Robots.txt helps the crawlers avoid the unimportant pages on your blog and rather concentrate on the important areas, thus boosting your SEO (Search Engine Optimisation) considerably.

How To Use The Custom Robots.txt Generator For Blogger?

Generate Custom Robot.txt

Simply enter your Blogger site’s URL along with https:// and www. in the above tool and press the “Generate Robots.txt” button.

Now hit the “Copy” button. The Robots.txt code for you Blogger blog will now be copied in your clipboard.

Submit Custom Robot.txt To Blogger

Now login to your Blogger account.

Then go to Settings and scroll down to Crawlers and indexing (NEW INTERFACE)

Turn on “Enable custom robots.txt” and click on “Custom robots.txt“

Now paste the robots.txt code using CTRL+V on Windows or Command+C on Mac.

Hit the “Save” button.

OR

Go to Settings -> Search Preferences -> Crawlers and indexing. (OLD INTERFACE)

And click on Edit

Now set “Enable custom robots.txt content?” to “Yes“.

Paste the robots.txt code that you copied from my Blogger Robots.txt generator tool, inside the box and click on “Save changes“.

That’s it!

Congratulations!

You have successfully submitted the custom robots.txt file to your Blogger blog!

Custom Robots Header Tags

Along with custom robots.txt, Blogger also provides you with the option to set specific rules for web crawlers for individual page types (homepage, archive, search, posts and pages) with custom robots header tags.

Here’s how you can setup the Custom Robots Header Tags with the optimal settings:

NEW INTERFACE

Turn on the “Enable custom robots header tags” option, which you will find just under the “Custom robots.txt” option.

Then click on Home page tags and configure it as shown below in the screenshot.

Hit the SAVE button.

Next click on “Archive and search page tags” and configure it as shown below and hit SAVE.

Finally, configure the “Post and page tags” and save it.

OLD INTERFACE

In the old (legacy) interface of Blogger, you will find the “Custom robots header tags” just after the “Custom robots.txt” option.

Click on the edit button next to the “Custom robots header tags” option and configure it as shown in the screenshot below:

Click on Save changes.

That’s It!

Don’t forget to share this article on your favourite social media if you liked it. Feel free to comment down below if you have any confusions or doubts.

Talk to you again, soon. Have a great day ahead!