PonyMirrors/philomena
1
0
Fork 0
mirror of https://github.com/philomena-dev/philomena.git synced 2024-11-23 20:18:00 +01:00
Code Issues Projects Releases Packages Wiki Activity

search engine spiders should do their own searching

Browse source
This commit is contained in:
byte[] 2020-08-15 16:49:27 -04:00
parent aef3cd399b
commit 7046857cbd
1 changed files with 6 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
assets/static/robots.txt
View file

@ -1,6 +1,11 @@
User-agent: * User-agent: *
Crawl-delay: 0.5 Crawl-delay: 2
Allow: /images/*
Disallow: /images
Disallow: /tags
Disallow: /search
Disallow: /profiles Disallow: /profiles
Disallow: /api Disallow: /api