danbarber/danbarber.me.hugo
1
0
Fork 0
mirror of https://github.com/danbee/danbarber.me.hugo.git synced 2025-03-04 08:59:18 +00:00
Code Issues Releases Wiki Activity

Block AI from scaping my site

Browse Source
This commit is contained in:
Daniel Barber 2024-03-20 21:47:47 -05:00
parent 0b26784130
commit edbf2ffdfd
1 changed files with 68 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
static/robots.txt Normal file
View File

@ -0,0 +1,68 @@
User-agent: AdsBot-Google
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: omgili
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: peer39_crawler
Disallow: /
User-agent: peer39_crawler/1.0
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: YouBot
Disallow: /