This file prevents web spiders/crawlers such as Googlebot from accessing all or parts of your website which is publically viewable. Related Web Spider Crawl
