# robots.txt file # # Follows the same syntax as a normal robots exclusion file. It should be # placed in the root directory that you are going to index, and will stop the # indexer looking at the files below the specified directories. User-agent: * Disallow: /common/media Disallow: /site/media