View unanswered posts | View active topics It is currently Wed Jun 19, 2024 3:21 pm



← Back to the Calcudoku puzzle page




Reply to topic  [ 3 posts ] 
 recent slowness and errors 
Author Message
User avatar

Posted on: Sat Apr 27, 2024 12:56 pm




Posts: 3323
Joined: Thu May 12, 2011 11:58 pm
 recent slowness and errors
I first got warnings about a very high load on the server yesterday morning, same thing today.
This while the number of users on the site is fairly constant.

After looking into this a bit, I found that the "Claudebot" is to blame: the crawler software of "Anthropic" (www.anthropic.com).

I've seen an increase in AI crawlers recently (everybody needs to amass tons of data to train their AI chatbots), but this one is pretty extreme,
by far the most aggressive crawler ever. It also seems to ignore explicit instructions to not crawl the site (using a commonly accepted
method for asking crawlers not to crawl the site).

(see for example: https://www.reddit.com/r/Anthropic/comm ... robotstxt/ and
https://www.reddit.com/r/singularity/co ... aping_the/).

I know it's important that these companies steal all data they can get their hands on so it can be processed and sold,
but I'm not very happy about it overloading my site [mellow]


Profile
User avatar

Posted on: Sat Apr 27, 2024 1:10 pm




Posts: 3323
Joined: Thu May 12, 2011 11:58 pm
Post Re: recent slowness and errors
.. sent them a note:

https://twitter.com/calcudokudotorg/sta ... 5607201845


apparently it's all over the net:

https://twitter.com/search?q=claudebot&src=typed_query


Profile

Posted on: Sat Apr 27, 2024 3:48 pm




Posts: 257
Joined: Tue May 24, 2011 4:55 pm
Post Re: recent slowness and errors
pnm wrote:

Gotta hope the backlash will make them reconsider their approach.


Profile
Display posts from previous:  Sort by  
Reply to topic   [ 3 posts ] 

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  
All forum contents © Patrick Min, and by the post authors.

Forum software phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by STSoftware.