#Tells web spiders which directories to avoid User-agent: * Disallow: /awstats/ #not sure if the following is correct syntax: Disallow: *.cgi