How to add custom Robots.txt file in Blogger
If you just starting to know about Search Engine
optimization, then there is a little chance that you might have heard about the
importance of robots.txt file and how it works in conjunction with the sitemap
of a site or blog. That is why in today’s article, I’m going to show you, how to add a custom robots.txt file inblogger. In our previous article, we have discussed about how to set up
or customize robots header tags in blogger. If you haven’t read that already,
then i suggest you to do that now and return here to know about adding a custom
robots.txt file in blogger. So here we go.
What is Robots.txt file ?
Before going into the actual details of adding a custom
robots.txt, you need to know about what a robots.txt file is and what is it
used for ? And before doing that, you need to know about the Sitemap file of a
website. A Sitemap file of a website tells the crawlers and search engines to
index their site and it’s updates in the search results. It is one of most
essential and required thing for a website. You can set that up easily for your
site, if you are using Google Webmasters tool. If you are a not sure what
Google Webmasters tool is, then please read our guide called as Google Webmasters Tools 101 : Everything
you need to know about Google Webmasters. So a Sitemap tells the search engines and
crawlers to index their site, whereas the robots.txt file will tell the search
engines and crawlers to avoid or no-index which parts of their site or blog.
It’s that simple. But in the robots.txt file, we need to add the sitemap url
also. If you want an example of a robots.txt file then see the code below.
User-agent:
Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap:
http://example.blogspot.com/feeds/posts/default?orderby=UPDATED
Did you notice the sitemap option on the above code. We need
to place the example url with url of your sitemap. Now, let’s discuss about the
each part of the above code in detail. You need to understand about them, if
you want better results.
1. User-agent:
Mediapartners-Google :
Even though this part is related to Google adsense, I’d
recommend you to leave this part as it is, because this helps the adsense
crawlers to know about which ads to serve on your site. But usually all the
site will have another code, that is ;
2. User-agent: *
This code allows all the crawlers to start crawling and
index certain parts of your sites.
3. Disallow: /search
This tells the robots or crawlers to ignore all the links of
the site that are starting with keyword ‘search'. For example this link www.example.com/search?how-to-start-a-blog
will be ignored because it is having search at it’s end.
4. Allow: /
This code means allowing the crawlers and robots to index
the homepage.
If you want to disable particular post from being indexed
then you need to use the code below.
Disallow: /2013/03/post-url.html
Well the last part is the sitemap and we have already
discussed about it. I think that simple explanation has cleared your doubts
regarding robots.txt. So, now let’s move onto our main topic, how to add custom
robots.txt file in blogger. Below are the steps that I’ve mentioned in detail
about adding a custom robots.txt file in blogger in detail. Please read them
carefully and do the same for your site or blog.
Steps to add custom robots.txt file in Blogger
Step 1:
Login to your blogger dashboard.
Step 2:
Navigate or go to Settings >> Search Preferences ››
Crawlers and indexing ›› Custom robots.txt ›› Edit ›› Yes.
Step 3:
Now add the custom robots.txt file in the box and click
save.
That’s all there it is. You may also want to know about how
to generate a sitemap for your blogger blog. Anyway, hope you like this
article. Please share your views about this article in the comment section
below. Thank you.