Learn seo online!

what is robot file

As for seo, robots.txt is a very important file.  It instruct  spider  to crawl your website content. When spider come to your website, if your website robots.txt is not exist, lots of  content will be crawled including privacy content. So getting a robots.txt for your website is very important, it tells spider which content in your website should not be crawled , and which content should be crawled. I met some website owners , they asked  in forum why their website content had not be indexed by google for a long time , i just check its robots.txt:  User-agent: *Disallow: /       , so of course when spider read this kind  robots.txt file , they will not crawl your website content.

how to write a robots.txt

usually the commands are like this

1  allow all robots to crawled your website content

User-agent: *
Disallow:

2  disallow all spider to crawled your website content

User-agent: *
Disallow: /

3  disallow spider to crawled some of your website content

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /tmp/
Disallow: /private/

you can see allow command in many robots.txt file , but right now not all the spider know allow command . For most of us , use disallow command is enough(i am a little lazy).

you also can put your robot in your page code like this:

<meta name="robots" content="noindex,nofollow" />

check your robot file 

check your robot file