Bot hitting Cloudflare block since today

Hi, I run a bot which has started getting blocked by Cloudflare.

I know bots don't necessarily have the best reputation these days, but I like to believe my bot is reasonably well-behaved.
It sets a custom user-agent with contact information, has a rate-limit and exponential back-off on errors.
It's been running fine for a while, but since a few hours ago it consistently hits the "Sorry, you have been blocked", "You are unable to access donmai.us" wall from CF.

Is there something else I should do to prevent it from being blocked or is there a particular reason for it being blocked?

I imagine it has to do with topic #31933 / everyone else getting hit with sporadic 502 errors due to bot activity today.

I totally agree that it's related, I just also happen to be on the on the side whose bot is now getting blocked, which I figured is a separate topic from the 502's hitting the end users.

If it's because my particular bot is sending too many requests I can lower the rate-limit further, though it is already significantly below the documented "global rate limit on read requests of 10 requests per second" (It only does reads).

same here! my little script to get background image for vscode from danbooru api is now getting 403 "Sorry, you have been blocked" from the cf when trying to fetch /posts.json. Tried setting useragent to my username/adding api key but i still cant access it :(

I enabled some stricter rules to try to block more AI scrapers, but it looks like it's blocking too many innocent bots. I've dialed it back down for now.

For reference we get a huge amount of bot traffic and it's a constant struggle trying to block scrapers from scraping the site to death while still allowing benign bots.

That seems to have fixed it for me, thanks.

I'm aware of the bot scraping situation. I'm sure you've already tried all kinds of solutions, so the only thing I can say is that it sucks and, as unlikely as it is, I hope people get tired of scraping for AI sooner rather than later.

evazion said:

I enabled some stricter rules to try to block more AI scrapers, but it looks like it's blocking too many innocent bots. I've dialed it back down for now.

For reference we get a huge amount of bot traffic and it's a constant struggle trying to block scrapers from scraping the site to death while still allowing benign bots.

Thank you!