Blogging Blogger (BlogSpot) Tips Custom Robots.txt Generator For Blogger (BlogSpot) Blog

Custom Robots.txt Generator For Blogger (BlogSpot) Blog

-

Blogger Robots.txt Generator Tool

Works with both, .blogspot.com and custom domains.

Enter your Blogger blog’s address (URL) below. [Including https:// and www. (if custom domain)]




What Is Robots.txt And How Is It Useful For Blogger (BlogSpot)?

Robots.txt is a text file which helps the bots/crawlers of the search engines such as Google and Bing to crawl and index your site effectively.

It is a set of instructions or rules, which the crawlers and search engine bots visiting your site, follow.

The Robots.txt file is also used by crawlers from other types of websites such as social media sites like Facebook, Twitter and SEO sites like online keyword research tools.

The main function of the Robots.txt file is to guide the crawlers and the bots, about which portion of your site should be available on the search engines and which parts should not show up on the search.

For example, a blogger would want all her/his articles to show up on the search engine, but would definitely not want other pages like the blog’s internal search pages to show up on Google!

Let’s consider a scenario:

Suppose a person visits your website and uses the search bar on your site to look up a random word unrelated to the niche of your website.

He/she might end up with something like this:

And in Blogger, the URL of that page will be: https://www.example.com/search?q=hallelujah

What will happen next is, the crawler in the background will most probably crawl this search results page as well and put it in its index. And due to that, these low-quality pages that will eventually get stored in Google’s index will affect the authority of the blog.

Here’s an example:

But with Robots.txt file present, it will instruct the web crawlers not to put such pages in their index and thus, preventing the indexing of such low-quality pages.

Thus, Robots.txt helps the crawlers avoid the unimportant pages on your blog and rather concentrate on the important areas, thus boosting your SEO (Search Engine Optimisation) considerably.

How To Use The Custom Robots.txt Generator For Blogger?

Generate Custom Robot.txt

Simply enter your Blogger site’s URL along with https:// and www. in the above tool and press the “Generate Robots.txt” button.

Blogger Custom Robots.txt Generator Tool

Now hit the “Copy” button. The Robots.txt code for you Blogger blog will now be copied in your clipboard.

Submit Custom Robot.txt To Blogger

Now login to your Blogger account.

Then go to Settings and scroll down to Crawlers and indexing (NEW INTERFACE)

Turn on “Enable custom robots.txt” and click on “Custom robots.txt

Now paste the robots.txt code using CTRL+V on Windows or Command+C on Mac.

Hit the “Save” button.

Submit custom robots.txt to blogger

OR

Go to Settings -> Search Preferences -> Crawlers and indexing. (OLD INTERFACE)

And click on Edit

Submit Custom Robots.txt in Old Interface of Blogger

Now set “Enable custom robots.txt content?” to “Yes“.

Paste the robots.txt code that you copied from my Blogger Robots.txt generator tool, inside the box and click on “Save changes“.

enable custom robots.txt and paste the code.

That’s it!

Congratulations!

You have successfully submitted the custom robots.txt file to your Blogger blog!

Custom Robots Header Tags

Along with custom robots.txt, Blogger also provides you with the option to set specific rules for web crawlers for individual page types (homepage, archive, search, posts and pages) with custom robots header tags.

Here’s how you can setup the Custom Robots Header Tags with the optimal settings:

NEW INTERFACE

Turn on the “Enable custom robots header tags” option, which you will find just under the “Custom robots.txt” option.

Then click on Home page tags and configure it as shown below in the screenshot.

custom robot tags for home page

Hit the SAVE button.

Next click on “Archive and search page tags” and configure it as shown below and hit SAVE.

custom robot tags for archive and search pages

Finally, configure the “Post and page tags” and save it.

custom robot tags for posts and pages

OLD INTERFACE

In the old (legacy) interface of Blogger, you will find the Custom robots header tags” just after the “Custom robots.txt” option.

Click on the edit button next to the “Custom robots header tags” option and configure it as shown in the screenshot below:

custom robots header tags blogger old interface

Click on Save changes.

That’s It!

Don’t forget to share this article on your favourite social media if you liked it. Feel free to comment down below if you have any confusions or doubts.

Talk to you soon again. Have a great day ahead!

Anirban Roy
Anirban Roy is the founder of TechRBun. He is a Computer Geek. He likes to share his knowledge about PC, Mobiles and Blogging. He studies in class XII and when he is not studying, he can always be found tweaking his PC or surfing the web on his mobile phone. He loves music and literature too!

5 COMMENTS

  1. Great website and wonderful information. God bless you. I just shared your page and that deserves it to be spread to all. thanks.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

You might also likeRELATED
Recommended to you