Securing pages from Bots

[gotFusion]

Read my robots.txt tutorial or google for others.

You can restrict doc and file types as well as entire pages or folders.

If you want to be ubler stealthy, you can add a restriction to the way back machine so that even the archive bot does not store your content.

https://web.archive.org/web/*/http://www.gotfusion.com

[Reacher]

You should have a single robots.txt file in your domain root and a meta robots line in the doc head for the pages you wish to disallow crawling.

They are not the same thing so use them both.

Look at your stats. you should see 404 listings for the bots that call robots.txt upon visiting the domain. All well behaved bots will look at the robots.txt file before starting the crawl as it tells them what to index and what not to index.

I have made a robots.txt file from the example on the gotfusion tut. I have a couple of questions.

(1) If I specify DISALLOW a folder, does this mean that all sub-folders and pages in that folder are included in the disallow.

(2) I found a robots.txt file in my public-html directory that says "sitemap.xml", that NOF put there by default. I assume this file means that bots can crawl everything on the site. Should I replace this one with my new robots.txt file that includes the "DISALLOW: /sitemap.xml" file? ...and should I leave the existing sitemap.xml there?