From 0baa8ad58779d12275eec3eec47e976a413f57be Mon Sep 17 00:00:00 2001 From: Daniel Barber Date: Thu, 13 Jun 2024 13:56:28 -0500 Subject: [PATCH] Update robots.txt Add Applebot-Extended so Apple doesn't crawl my posts to train LLMs --- static/robots.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/static/robots.txt b/static/robots.txt index b1f84604..4bd0685f 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -7,6 +7,9 @@ Disallow: / User-agent: anthropic-ai Disallow: / +User-agent: Applebot-Extended +Disallow: / + User-agent: AwarioRssBot Disallow: /