Server load problems

[antus]

https://forum.level1techs.com/t/small-g ... /233569/11
Capture.PNG

We had similar issues over on the JC forum, Darren had to set it up so CLoudfare would challenge and verify all users or the server was getting overloaded.

This triggered a penny drop moment for me. I've been reading and learning a lot the last couple of days. I thought cloudflare applied challenges as needed or blocks by default, but it seems it does not. Which does make sense else you'd be screwed in case you hit false positives. I need to set up a managed challenge rule that matches the whole world. Then modify the known bad data centre rules from managed challenge to outright block and put them first in the rule list. Lets see how that goes....

[antus]

For anyone who finds this thread with google and is fighting similar problems elsewhere, the magic rule to make cloudflare challenge everyone to the required level automatically like you expect it to do out of the box is a WAF custom rule (or security rule depending old or new cloudflare UI) of

Code: Select all

(ip.src in {0.0.0.0/0}) or (ip.src in {2000::/3})

set to managed challenge. Only then do you really have the protection in place. Without it, you have just a proxy and a cache.

[antus]

This is looking promising. Its only been on for 30 minutes and its challenged over 100,000 connections with only 0.05% passing the challenge. The rest were bots! eek. But also, good. Database load is down to 2% of a possible 800%, which it was hitting at last nights peak.

[antus]

updated the rule to be:

Code: Select all

Expression: ((ip.src in {0.0.0.0/0}) or (ip.src in {2000::/3})) and not cf.client.bot

As I would have shot myself in the foot and blocked the site from search engines too, effectively as good as wiping us off the searchable internet.

the "and not cf.client.bot" is the ticket, and the right brackets to control how it is parsed and processed.

[AngelMarc]

So no hardware upgrade needed? Or that still happening anyway?

[antus]

Well its all bought and on the way, so I might as well follow through. There will be more than the current user count within the 24 hour cycle and peaks and troughs as software is updated and announced on facebook groups etc but I probably could have gotten away with the original system for longer. Oh well that's how it works. You can only act on what you know and this site is all about sharing technical knowledge and the learnings from the journey. The old server was too small even if it was copeing, so its not a bad thing to have solved that and greater capability. Now I know it will be good for 150,000 legit users in a day if it ever reaches that for real and it seems cloudflare free with the right configuration is a valid answer.

It is a bit sad to learn 99.95% of the traffic was fake, i'd rather at least a couple of percent were real, but hey...

[AngelMarc]

Makes sense. So do you build it and install it in a datacenter and just rent bandwidth?
I've also been wondering how many of my ECU code downloads were bot/AI traffic.

[antus]

I'll tell you via PM. I try to balance what I share incase its used against us.

I cant really tell you what was bots, because I have no way of identifying them in retrospect, only blocking them going forward. I'd think they were mostly AI bots scraping posts only, not attachments, but I cant back that up with stats. I never had an observable problem with downloads, mainly database when them scrapeing too quick and clicking on expensive links to generate frequently which humans rarely click.

I think the cloudflare cache is working a lot better now which is actually hideing the true number of guests and its higher than what is shown. I am tempted to turn the cache off entirely so the numbers are real. That 2.23% is missleading because its the last 24 hours, so the number will climb for the next 22 or so until we have a 24 hour stat with the same configuration. Look at the dot point top right to see the difference. Or maybe I turn off the list of users as it is now misleading.

[AngelMarc]

As I expected, you know plenty more about this subject than I do.
The most I've done with a server is following instruction to set up a Palworld server, which I only used myself. Kept the game running for game resource collection without it requiring GPU load. Was disappointed that even on the same machine as client, with no guests, I was getting the same laggy/stuttery non-sense as public servers.
Those are numbers I know I shouldn't care about, but like having. Just like with YouTube videos. I occasionally sort by popularity and see what's what, and check like count on videos I have more personal interest in. Won't make or break the usefulness of the forum.

[antus]

Things appear stable, and user count is about where i'd expect it.