I first got warnings about a very high load on the server yesterday morning, same thing today.
This while the number of users on the site is fairly constant.

After looking into this a bit, I found that the "Claudebot" is to blame: the crawler software of "Anthropic" (www.anthropic.com).

I've seen an increase in AI crawlers recently (everybody needs to amass tons of data to train their AI chatbots), but this one is pretty extreme,
by far the most aggressive crawler ever. It also seems to ignore explicit instructions to not crawl the site (using a commonly accepted
method for asking crawlers not to crawl the site).

(see for example: https://www.reddit.com/r/Anthropic/comm ... robotstxt/ and
https://www.reddit.com/r/singularity/co ... aping_the/).

I know it's important that these companies steal all data they can get their hands on so it can be processed and sold,
but I'm not very happy about it overloading my site

.. sent them a note:

https://twitter.com/calcudokudotorg/sta ... 5607201845


apparently it's all over the net:

https://twitter.com/search?q=claudebot&src=typed_query

Gotta hope the backlash will make them reconsider their approach.