Quick Guide To Setup and Optimized ''Robots.Txt'' In Blogger

Have you ever heard the name robots.txt or have you added your own custom robots.txt file in your Blogger blog. Here we will discuss about what robots.txt is and how to create and add custom robots.txt file in Blogger. All that deals with crawling and indexing of your blog which covers your blog SEO, so take note on this article.

In Blogger you have a section called search preferences that covers your blogs Meta tags, errors /redirection & crawling and indexing. There you can manage your blogs view with search engines also to maintain a healthy blog you need to customize your blogs search preferences effectively.

Many of beginners blog after setup their blog they need to manage their blog and made its search preference very well,in such case they need to crawal and index their site.Thus they need few things inculding robots.txt.

The robots.txt are further divided into two sub categories i.e custom robots.txt and custom robots header tags,her we put both categories,thus we name it "Quick Guide to Robots.Txt".Let’s see about robots.txt and the use of adding custom robots.txt file in Blogger,but first learn what is robots.txt;

What Is robots.txt ?

There is a hidden, relentless force that permeates the web and its billions of web pages and files, unbeknownst to the majority of us sentient beings. I'm talking about search engine crawlers and robots here. Every day hundreds of them go out and scour the web, whether it's Google trying to index the entire web, or a spam bot collecting any email address it could find for less than honorable intentions.

As site owners, what little control we have over what robots are allowed to do when they visit our sites exist in a magical little file called "robots.txt."

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

Adding Robots.Txt File In Blogger:

Each Blogger blog will have a robots.txt file that comes by default and it looks something like the one below. You can check your blogs robots.txt file by adding /robots.txt next to your domain name. (http://yoursite.blogspot.com/robots.txt)

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://yoursite.blogspot.com/feeds/posts/default?orderby=UPDATED

Explanation:

As you can see above the default robots.txt file has few things like user-agent, media partners-Google, user-agent:*, disallow and site map. If you are not aware about these, then here is the explanation.

First you need to know about User agent which is a software agent or client software that will act on behalf of you.

Mediapartners-Google: Media partner Google is the user agent for Google adsense that is used to server better relevant ads on your site based on your content. So if you disallow this they you will won’t able to see any ads on your blocked pages.

User-agent:* – So you all know what user-agent is, so what is user-agent:*. The user-agent that is marked with (*) asterisk is applicable to all crawlers and robots that can be Bing robots, affiliate crawlers or any client software it can be.

Disallow: By adding disallow you are telling robots not to crawl and index the pages. So below the user-agent:* you can see Disallow: /search which means you are disallowing your blogs search results by default. You are disallowing crawlers in to the directory /search that comes next after your domain name. That is a search page like http://yoursite.blogspot.com/search/label/yourlabel will not be crawled and never be indexed.

Allow – Allow: / simply refers to or you are specifically allowing search engines to crawl those pages.

Sitemap: Sitemap helps to crawl and index all your accessible pages and so in default robots.txt you can see that your blog specifically allowing crawlers in to sitemaps. You can learn more about Blogger sitemap here. There is an issue with default Blogger sitemap, so learn how to create sitemap in Blogger and notify search engines.

Best and recommended robots.txt file for Blogger:

Only use custom robots.txt file if you are 100% sure on what you are doing. Improper use of custom robots.txt can harm your site rankings. So for best results it is recommended that you use default robots.txt in Blogger which works good. But change the default sitemap in your robots.txt and add your custom sitemap for Blogger.

How to add custom robots.txt file in Blogger:

Go to your blogger blog dashboard.

Navigate to Settings >> Search Preferences ›› Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes

Now paste your robots.txt file code in the box.

Click on Save Changes button.

You are done!

Once done click save changes. Now to check your robots.txt just add /robots.txt at the end of your blog URL and you can see your custom robots.txt file. After adding your custom robots.txt file you can submit your blog to search engines.

How To Setup Custom Robots Header Setting In Blogger:

Custom robots header tags allow you to tell web crawlers not to index certain sections of your site. It’s similar to customrobots.txt, but not as granular. For example, if you check the box for "nofollow" under the "Homepage" section, it means that you don’t want web crawlers to consider any links on your homepage leading to any other page.

Thee three sections you’ll see below correspond to:

Your blog’s home page

Your archive and search pages

Your posts and pages

Check the boxes to set the flags according to your preferences, then view the sections of your blog that you customized (home page, archive and search pages, pages and posts). You should see an X-Robots-Tag header tag in the HTTP response with the selected flags.

Now,click on save changes button,and you have done.

Video Tutorial:

From The Editor’s Desk:

Hope this post clearly explained you about robots.txt and how to create and add custom robots.txt file in Blogger. Please share it and if you have any other doubts on Blogger robots.txt then feel free to comment below.