in , ,

Custom Robots.txt Generator For Blogger (BlogSpot) Blog

Blogger Robots.txt Generator Tool

Works with both, .blogspot.com and custom domains.

Bliss - The Best Blogger (Blogspot) Template

Enter your Blogger blog’s address (URL) below. [Including https:// and www. (if custom domain)]




What Is Robots.txt And How Is It Useful For Blogger (BlogSpot)?

Robots.txt is a text file which helps the bots/crawlers of the search engines such as Google and Bing to crawl and index your site effectively.

It is a set of instructions or rules, which the crawlers and search engine bots visiting your site, follow.

The Robots.txt file is also used by crawlers from other types of websites such as social media sites like Facebook, Twitter and SEO sites like online keyword research tools.

The main function of the Robots.txt file is to guide the crawlers and the bots, about which portion of your site should be available on the search engines and which parts should not show up on the search.

For example, a blogger would want all her/his articles to show up on the search engine, but would definitely not want other pages like the blog’s internal search pages to show up on Google!

Let’s consider a scenario:

Suppose a person visits your website and uses the search bar on your site to look up a random word unrelated to the niche of your website.

He/she might end up with something like this:

And in Blogger, the URL of that page will be: https://www.example.com/search?q=hallelujah

What will happen next is, the crawler in the background will most probably crawl this search results page as well and put it in its index. And due to that, these low-quality pages that will eventually get stored in Google’s index will affect the authority of the blog.

Here’s an example:

But with Robots.txt file present, it will instruct the web crawlers not to put such pages in their index and thus, preventing the indexing of such low-quality pages.

Thus, Robots.txt helps the crawlers avoid the unimportant pages on your blog and rather concentrate on the important areas, thus boosting your SEO (Search Engine Optimisation) considerably.

Bliss - The Best Blogger (Blogspot) Template

How To Use The Custom Robots.txt Generator For Blogger?

Generate Custom Robot.txt

Simply enter your Blogger site’s URL along with https:// and www. in the above tool and press the “Generate Robots.txt” button.

Blogger Custom Robots.txt Generator Tool

Now hit the “Copy” button. The Robots.txt code for you Blogger blog will now be copied in your clipboard.

Submit Custom Robot.txt To Blogger

Now login to your Blogger account.

Then go to Settings and scroll down to Crawlers and indexing (NEW INTERFACE)

Turn on “Enable custom robots.txt” and click on “Custom robots.txt

Now paste the robots.txt code using CTRL+V on Windows or Command+C on Mac.

Hit the “Save” button.

Submit custom robots.txt to blogger

OR

Go to Settings -> Search Preferences -> Crawlers and indexing. (OLD INTERFACE)

And click on Edit

Submit Custom Robots.txt in Old Interface of Blogger

Now set “Enable custom robots.txt content?” to “Yes“.

Paste the robots.txt code that you copied from my Blogger Robots.txt generator tool, inside the box and click on “Save changes“.

enable custom robots.txt and paste the code.

That’s it!

Congratulations!

You have successfully submitted the custom robots.txt file to your Blogger blog!

Custom Robots Header Tags

Along with custom robots.txt, Blogger also provides you with the option to set specific rules for web crawlers for individual page types (homepage, archive, search, posts and pages) with custom robots header tags.

Here’s how you can setup the Custom Robots Header Tags with the optimal settings:

NEW INTERFACE

Turn on the “Enable custom robots header tags” option, which you will find just under the “Custom robots.txt” option.

Then click on Home page tags and configure it as shown below in the screenshot.

custom robot tags for home page

Hit the SAVE button.

Next click on “Archive and search page tags” and configure it as shown below and hit SAVE.

custom robot tags for archive and search pages

Finally, configure the “Post and page tags” and save it.

custom robot tags for posts and pages

OLD INTERFACE

In the old (legacy) interface of Blogger, you will find the Custom robots header tags” just after the “Custom robots.txt” option.

Click on the edit button next to the “Custom robots header tags” option and configure it as shown in the screenshot below:

custom robots header tags blogger old interface

Click on Save changes.

That’s It!

Bliss - The Best Blogger (Blogspot) Template

Don’t forget to share this article on your favourite social media if you liked it. Feel free to comment down below if you have any confusions or doubts.

Talk to you again, soon. Have a great day ahead!

What do you think?

108 Points
Upvote Downvote

Written by Anirban Roy

Anirban is a full-stack developer and an SEO expert. He has 6 years of experience in web development and software engineering. With a passion for writing, he has been blogging since 2017.

Leave a Reply

Your email address will not be published. Required fields are marked *

19 Comments

  1. Thanks for this. I have applied it to my blog.

    But I wanted to ask: In my generated Robots.txt above the sitemap URL ends with two forward slashes.

    Like this: //sitemap.xml

    Is that correct? Or a mistake with your generator?

    Cheers.

    • There should be only one forward slash. This might have happened because you had included the trailing slash in the URL input box. Thanks for pointing out this issue. I will modify the generator to take care of this kind of issues in the future. For now, kindly use just a single slash, that is, /sitemap.xml