What is robot.txt? Syntax of robot.txt?

What is robot.txt?

This is a file for in the website some of the folder we dont want to crawl by the search engine optimization, suppose in the images folder we dont want to crawl .

The syntax for the robot.txt

****************

User-agent:

Disallow:

****************

User- agent  : This is for Search engine name

Disallow : This is for the folder name

eg:

User-agent : Googlebot

Disallow:/images/

then google search engine does not crawling the images folder of site.

For all search engines does not want to crawling then use the following code:

User-agent : *

Disallow: /images/

Disaallow : /admin/

Hope this will be useful …

You may like these posts