User-agent: Browsershots Disallow: User-agent: panscient Disallow: / # disallow all files in these directories User-agent: * Disallow: /notfound.shtml Disallow: /cgi-bin/ Disallow: /inc/ Disallow: /stats/ Disallow: /logs ###############################