Slowness today

Sorry about the slowness today guys, was busy IRL but noticed the overall slowdown on my phone yikes.
Not sure what went wrong, I think I had forgotten to empty some DB tables or something.

Not related to peak traffic (we saw 3700+ on the weekend and the site was running pretty good minus the odd hiccup).
Not related to API usage which has only increased slightly.

Made some small arrangements to the furniture and did a bit of dusting just now + a restart.
Let me know how it feels next 24hrs?

Teeb

For the DC tracker, seems it reverted to an older version, displaying the 'find game' icon instead of reporter id.

Bisu wrote: 2 years ago

For the DC tracker, seems it reverted to an older version, displaying the 'find game' icon instead of reporter id.

Thanks, fixed.

It seems a lot faster now than it was during the daytime. Things seem to be working well now.

The site was very slow between 19:00 - 20:00 today, somewhat coincide with the NA ladder DC walk.

Wonder if there was a DDoS attack or something.

It definitely seemed like it? Got error 503 a few times

Very much felt like it. It literally started the second SoJs started being sold. Took out the API response for about 5 minutes as well.

Not sure if that's an actual "saboteur" or just insane natural traffic upon walks.

Thanks for the reports guys, was asleep. Could someone give an estimate of how many people the site was showing as online at the time? Just so I can correlate with my own analytics and see where pain points are in the logs.

Teebling wrote: 2 years ago

Thanks for the reports guys, was asleep. Could someone give an estimate of how many people the site was showing as online at the time? Just so I can correlate with my own analytics and see where pain points are in the logs.

Sadly, no idea. Actually had to go double-check where to even find that just now. :/

If time helps though:
It started at pretty much exactly 4am CEST (which is also when the SoJ selling started), kept the API broken (non-responsive with a 30s timeout window at least) until about 4:10am CEST and continued to be significantly slowed with occasional API failures until about 4:40-4:45am CEST at which point the spawn was done and things returned to normal.

Schnorki wrote: 2 years ago

Not sure if that's an actual "saboteur" or just insane natural traffic upon walks.

... if I would have to put a bet - second option.

Deiphos wrote: 2 years ago

Schnorki wrote: 2 years ago

Not sure if that's an actual "saboteur" or just insane natural traffic upon walks.

... if I would have to put a bet - second option.

Normally I'd agree and we've seen slowdowns on previous walks. But those were always slowly coming in and worsening as the first progress message popped up (probably due to tons of people suddenly spam-refreshing the tracker).

But this time we literally went from 0 to f***ed the second he started selling, before any progress message had hit the server yet.

Then again, at least Luciano's stream apparently also had record viewer numbers so I guess word got out to the point where there may have been enough people already here, ready to spam F5 (the question being why, seeing how they would then all already be on the server anyways).

Teebling wrote: 2 years ago

Thanks for the reports guys, was asleep. Could someone give an estimate of how many people the site was showing as online at the time? Just so I can correlate with my own analytics and see where pain points are in the logs.

Still getting some 524 Cloudflare errors sporadically.

Schnorki wrote: 2 years ago

Deiphos wrote: 2 years ago

Schnorki wrote: 2 years ago

Not sure if that's an actual "saboteur" or just insane natural traffic upon walks.

... if I would have to put a bet - second option.

Normally I'd agree and we've seen slowdowns on previous walks. But those were always slowly coming in and worsening as the first progress message popped up (probably due to tons of people suddenly spam-refreshing the tracker).

But this time we literally went from 0 to f***ed the second he started selling, before any progress message had hit the server yet.

Then again, at least Luciano's stream apparently also had record viewer numbers so I guess word got out to the point where there may have been enough people already here, ready to spam F5 (the question being why, seeing how they would then all already be on the server anyways).

Sorry - just saw that I messed up.
Wanted to agree to the Sabotage possibility.

Performance is going down again. :/
(<600 online)

Schnorki wrote: 2 years ago

Performance is going down again. :/
(<600 online)

Incorrect. <600 online is because I just purged the sessions table, logging everyone out. Back up to nearly 2k now since you posted, performance is alright.
I am monitoring the server all the time guys and troubleshooting issues. It's not easy.

Fair enough. More than 600 then. That's what you get for creating something that's actually good and popular!

Teebling wrote: 2 years ago

Schnorki wrote: 2 years ago

Performance is going down again. :/
(<600 online)

Incorrect. <600 online is because I just purged the sessions table, logging everyone out. Back up to nearly 2k now since you posted, performance is alright.
I am monitoring the server all the time guys and troubleshooting issues. It's not easy.

Don't fret friend - we know you are the dark knight with overwatch. All of us are here to always manage up feedback just to keep you apprised of what we are experiencing.

Cheers mate.

I've been getting massive slows and errors for the last ten minutes or so. Cloudflare 520, 522 and 524 all kept popping up. This is connecting from the Cloudflare Frankfurt servers.

Suddenly it stopped though right as I was writing this. It kinda seems odd? DDOS attempts? CF messing up? :O

Jinsho wrote: 2 years ago

Suddenly it stopped though right as I was writing this. It kinda seems odd? DDOS attempts? CF messing up? :O

Yeah, something is clogging the pipes at the moment and I don't know exactly what it is yet.

I restarted Apache + MySQL just before you wrote your post, which is why it started working again.

Think of it like covid - the site has a runny nose, it builds up until its congested, and I can fix that temporarily by blowing the nose (restarts etc.).

I know the site has covid or some other ailment, but I don't know what strain it is yet and its massively complicated to analyse and diagnose the root issue.

It could be automated malicious traffic, it could be a slow query or two, it could be a netstorm, it could be Cloudflare, it could be my server configuration, it could be a localised issue, it could be a bottleneck external resource, it could be a temp DB table filling up, it could be a hanging PHP process, it could literally be any of these things - you would think that it would be easy to spot where the problem is and fix it, but I must stress that it really is not.

The site isn't a simple blog page with some text and images, its a complicated engine built by an idiot (me) that requires a lot of attention to keep 2k concurrent online users interacting with each other.

As always, I am checking on the site throughout my working day, and doing everything I can in my power to keep the site running acceptably.

Yeah, I like the site. Thanks for making it!