Recent BOT & AI traffic has increased five fold on our forum

[Santeri]

The amount of web traffic this forum has received has increased five times in the past few months. Almost all bots were blocked from the server for many years and the block was removed in the beginning of this year. This was done to reduce the amount of automated link spam. Despite of having a separate forum for spammers, many of them continued spamming indiscriminately forums and posts with irrelevant porn links. That led to the removal of the spam forum and having all the new posts moderated manually, and returning this to all bots: Bots have not only increased the traffic but also the server load. So far both traffic and load remain OK so we have not taken any actions, but out of curiosity here is the list of all bots that have visited this forum today or in the past 2 weeks:

Active today

claudebot@anthropic.com (a very agressive AI bot responsible for over 70% of the web traffic and load)
PetalBot (formerly Aspiegelbot, also very aggressive made by Huawei, responsible for over 20% of all bot load and traffic)
Ahrefs
SeznamBot
Google
Bing
Majestic-12
Bytespider
DuckDuckBot
Trendictionbot
DotBot
coccocbot
SemrushBot
Applebot
Yandex
MSNbot
Amazonbot
Curl
GPTBot

Active within the past 2 weeks time

redditbot
ImagesiftBot
Baidu
SeekportBot
Google Feedfetcher

There has been altogether 15 bots more visiting this forum since 2017, but those have given up more recent crawling: W3C [Linkcheck], BLEXBot, W3C [Validator], Yahoo [Bot], Google Adsense [Bot], AdsBot [Google], Alexa [Bot], YaCy [Bot], Googlebot Smartphone, MSN [Bot], Ask Jeeves [Bot], FAST WebCrawler [Crawler], AspiegelBot, Voyager [Bot] and Exabot [Bot].

If the forum traffic or load becomes excessive, I will block them all again by returning them the good old http status 418: I'm a teapot.

Happy webmastering,

Santeri

[Santeri]

To mitigate higher bot/spider server load and traffic, I created a fix to phpBB forum Spiders/Robots setting that can be applied without changing the source code. So far it has reduced bot load 90% and traffic 50% to an acceptable level. So no teapot solution, yet.