# theworldwidewiki.com/blog/robots.txt
#
# The format and semantics of the "/robots.txt" file are as follows: 
# The file consists of one or more records separated by one or more 
# blank lines (terminated by CR,CR/NL, or NL). Each record contains 
# lines of the form "<field>:<optionalspace><value><optionalspace>".
# The field name is case insensitive. Comments can be included in 
# file using UNIX bourne shell conventions: the '#' character is used
# to indicate that preceding space (if any) and the remainder of the
# line up to the line termination is discarded. Lines containing only
# a comment are discarded completely, and therefore do not indicate a record boundary.
# The record starts with one or more User-agent lines, followed by one
# or more Disallow lines, as detailed below. Unrecognised headers are ignored.
#
# User-agent
# The value of this field is the name of the robot the record is describing access policy for. 
# If more than one User-agent field is present the record describes an identical access policy
# for more than one robot. At least one field needs to be present per record.
# The robot should be liberal in interpreting this field. A case insensitive 
# substring match of the name without version information is recommended.
# If the value is '*', the record describes the default access policy for 
# any robot that has not matched any of the other records. It is not allowed
# to have multiple such records in the "/robots.txt" file.
#
# Disallow
# The value of this field specifies a partial URL that is not to be visited. 
# This can be a full path, or a partial path; any URL that starts with this 
# value will not be retrieved. For example, 
# Disallow: /help disallows both /help.html and /help/index.html, whereas 
# Disallow: /help/ would disallow /help/index.html but allow /help.html. 
#
# Any empty value, indicates that all URLs can be retrieved. At least 
# one Disallow field needs to be present in a record.
# The presence of an empty "/robots.txt" file has no explicit associated semantics,
# it will be treated as if it was not present, i.e. all robots will consider themselves welcome.
#
# NOTE: each group of records MUST end with a blank line - including the last one!
#
User-agent: Mediapartners-Google*
Disallow: /conf/
Disallow: /cron/
Disallow: /htsrv/
Disallow: /inc/
Disallow: /locales/
Disallow: /media/
Disallow: /plugins/
Disallow: /rsc/
Disallow: /skins/
Disallow: /skins_adm/
Disallow: /xmlsrv/
Disallow: /_header.php
Disallow: /admin.php
Disallow: /summary.php
Disallow: /adm.php
Disallow: /adm

#
User-agent: Googlebot
Disallow: /conf/
Disallow: /cron/
Disallow: /htsrv/
Disallow: /inc/
Disallow: /locales/
Disallow: /media/
Disallow: /plugins/
Disallow: /rsc/
Disallow: /skins/
Disallow: /skins_adm/
Disallow: /xmlsrv/
Disallow: /_header.php
Disallow: /admin.php
Disallow: /summary.php
Disallow: /adm.php
Disallow: /adm

#
User-agent: *
Disallow: /conf/
Disallow: /cron/
Disallow: /htsrv/
Disallow: /inc/
Disallow: /locales/
Disallow: /media/
Disallow: /plugins/
Disallow: /rsc/
Disallow: /skins/
Disallow: /skins_adm/
Disallow: /xmlsrv/
Disallow: /_header.php
Disallow: /admin.php
Disallow: /summary.php
Disallow: /adm.php
Disallow: /adm