We Were Visited By 31 AI/Bots/Spiders/Crawlers Within 2 Days

Questions and discussion about web design, search engine optimisation and hosting
s
Posts: 336
Joined: 2017-7-5 09:58

Unread post by s » 2025-3-15 07:53

Today and yesterday there has been altogether 31 bots, crawlers and spiders visiting our free and open discussion forum. The amount of automated software and the bandwidth they consume has increased almost exponentially in the past years. The most recent, single biggest increase comes from AI bots.

We turned on the forum logs for a few days to find out where the increase is coming and used the logs for finding out all the automated visit we got in the past 2 days. Here is the list of visitors: We Were Visited By 31 AI/Bots/Spiders/Crawlers Within 2 Days moz.com SEO DotBot - 15.3.2025 08:50
Yandex - 15.3.2025 08:46
SemrushBot - 15.3.2025 08:42
Ahrefs - 15.3.2025 08:36
Google [Bot] - 15.3.2025 08:25
PetalBot - 15.3.2025 08:14
ImagesiftBot - 15.3.2025 08:02
GPTBot - 15.3.2025 08:00
ChatGPT - 15.3.2025 07:42
DuckDuckBot - 15.3.2025 07:15
Bing [Bot] - 15.3.2025 06:27
Majestic-12 [Bot] - 15.3.2025 06:17
redditbot - 15.3.2025 04:57
OpenAI Search Bot - 15.3.2025 04:48
Bytespider - 15.3.2025 03:51
Applebot - 15.3.2025 02:31
FacebookCrawler - 14.3.2025 23:07
Twitterbot - 14.3.2025 22:41
360Spider - 14.3.2025 21:15
Google Feedfetcher - 14.3.2025 21:15
PerplexityBot - 14.3.2025 16:55
claudebot - 14.3.2025 16:38
TikTokSpider - 14.3.2025 14:24
aiohttp - 14.3.2025 14:20
Barkrowler - 14.3.2025 10:31
coccocbot - 14.3.2025 10:24
Baidu Spider - 14.3.2025 09:28
mail.ru bot - 14.3.2025 08:48
Word Press bot - 14.3.2025 05:25
SeznamBot - 14.3.2025 03:22
CCBot - 14.3.2025 00:54

Happy hacking,

Santeri

PS. New bots appear and old change so fast that blocking them one by one is no longer an option. If you are running a phpBB forum, here are instructions how you can mitigate the increased automated traffic keeping your server performance high. The same principle is applicable to all dynamic websites. You can simply change how your website behaves instead of wasting your time trying to block all the automated software.