How to Write Robots.txt Text File

The robots.txt is a very simple text file that tells Search Engines, Internet robots and spiders which pages of your website are allowed or not allow to visit.

Where to Save Robots txt File
The Robots.txt file musr be saved on the home directory of your website, for example:

http://www.Domain.com/robots.txt.

Remember that only one robots.txt is allowed for one website.

For most websites, the Robots text files have only a few lines. Below are the most common examples of a Robot.txt file.

Robots txt File Example 1: Allow All Robots to Visit Anywhere

# Allows all robots to visit
# Allows them to spide anywhere of your website (i.e. Disallow none)

User-agent: *
Disallow:

Robots txt File Example 2: Allow All Robots to Visit Part of Website

# Allows all robots to visit, but
# Disallows visit the images directory and the cgi-bin directory.

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/

Robots txt File Example 3: Refuse All Robots to Visit

# Disallows all robots to visit

User-agent: *
Disallow: /

Robots.txt and Google Sitemaps

If you use Sitemaps for your website, the Robots [Sitemap] command should be included in your Robots.txt. Below is an example:

# Allows all robots to visit, but
# Disallows visit the images directory and the cgi-bin directory.
# I have a Sitemap, please visit it

User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Sitemap: http://www.Domain.com/sitemap.xml

This page discuss how to write a Robots.txt Text File