Setting the robots.txt file is important for your web site's health - Google uses it as a guide for what to index and what to exclude.

I've previously included references to the robots.txt file in this post on going live with a new elcomCMS site (see point 6) but it's worth covering again here.

In elcomCMS, setting the robots.txt file is really easy.

You can access it from the Admin menu:

and then edit it directly:

Recommended robots.txt Settings

Staging Site

If a site is in staging, you'll ideally want to block Google from crawling it. Use the following to block the entire site:

User-agent: *

Disallow: /

Live site

Once the site is live, you only really want to block a few items eg blocking the login page is reasonably common. You can also block entire directories eg such as an FTP folder:

User-agent: *

Disallow: /cmlogin.aspx

Disallow: /ftp/

Be very careful when going live with a site - I often see sites go live (from staging) and keep their robots.txt file from staging - and in the process block their entire live site in Google! Make sure you check your robots.txt file when going live.