philomena/assets/static/robots.txt

User-agent: *
Crawl-delay: 2

Allow: /images/*

Disallow: /images
Disallow: /tags
Disallow: /search
Disallow: /profiles
Disallow: /api

Disallow: /harming/humans
Disallow: /ignoring/human/orders
Disallow: /harm/to/self
robots config 2020-05-28 02:21:17 +02:00			`User-agent: *`
search engine spiders should do their own searching 2020-08-15 22:49:27 +02:00			`Crawl-delay: 2`
robots config 2020-05-28 02:21:17 +02:00
search engine spiders should do their own searching 2020-08-15 22:49:27 +02:00			`Allow: /images/*`

			`Disallow: /images`
			`Disallow: /tags`
			`Disallow: /search`
robots config 2020-05-28 02:21:17 +02:00			`Disallow: /profiles`
			`Disallow: /api`

			`Disallow: /harming/humans`
			`Disallow: /ignoring/human/orders`
			`Disallow: /harm/to/self`