what is robot.txt file in seo - Techno Pratap

what is robot.txt file in seo

What is robot.txt file in SEO


A robots.txt file is a plain text file that tells your web crawler what page to crawl and which to don't on your site. This is a very small text file located in the root directory of your server.
Almost all search engine bots comply with this robots.txt file. With this file, you can block any page of your site from the bot or block a particular type of file on your site such as GIF PNG.

also read:




In practice, it is suggested that the robots.txt file dictates which part of your website can be crowded and which part cannot be crowded.

An example of a robots.txt file is shown below.

robots.txt example


Now I will tell you about all the syntax of robots.txt here. You can learn about all the syntax used in robots.txt

robots.txt  syntax

Here the robots.txt syntax can be found as a language that instructs the robot or web crawler to know what to scan and what not to do on your site.

User-agent: The web crawler you order to crawl your site. The following is a list of most user-agents.

Disallow: "Disallow" is used to prevent the user-agent from crowding a specific URL or directory.

Allow:  {Only applicable for Googlebot} The syntax can inform a Googlebot that accesses a page or subfolder, although its parent page or subfolder is Disallowed.
Crawl-delay: How many seconds a crawler should wait before crawling another link. Googlebot does not acknowledge this syntax.

Sitemap: This is no command. Here you can paste your Sitemap URL.






Pratap Chandra Ghosh

Posted by Pratap Chandra Ghosh

Hi, I am Pratap, A Blogger & Digital Marketer. I am in the blogging field since August 2018, I love blogging by heart. And Also Love To Share knowledge On Social Platform.

Post a Comment

0 Comments