danbarber/danbarber.me.hugo
1
0
Fork 0
mirror of https://github.com/danbee/danbarber.me.hugo.git synced 2025-03-04 08:59:18 +00:00
Code Issues Releases Wiki Activity

Update robots.txt

Browse Source 
Add Applebot-Extended so Apple doesn't crawl my posts to train LLMs
This commit is contained in:
Daniel Barber 2024-06-13 13:56:28 -05:00 committed by GitHub
parent eabce39740
commit 0baa8ad587
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
static/robots.txt
View File

@ -7,6 +7,9 @@ Disallow: /
User-agent: anthropic-ai User-agent: anthropic-ai
Disallow: / Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: AwarioRssBot User-agent: AwarioRssBot
Disallow: / Disallow: /