Search the database
Search forum topics
Search members
Search for trades
diablo2.io is supported by ads
diablo2.io is supported by ads
128 replies   10987 views
2

Description

Description by Teebling
5

Can be used to make Runewords:

7
Don’t worry about slowdowns, a site this young undergoing constant changes in the form of feature additions, updating vs game patches, lots of “live” data, probably just lots of rows in general all while now under near-enterprise level traffic.

Agreed that it’s not worth diving into atm if reboots fix it — gotta let yourself relax sometimes! Mental health month and everything lol.
7
User avatar

EnragedN3wb 179Moderator

PC
Teebling wrote: 1 year ago
its a complicated engine built by an idiot (me)
The Server
The Server
7
OP
User avatar

Teebling 5763Admin

Europe PC
Image

7
User avatar

Schnorki 3024Moderator

PC
Sadly, the site got completely annihilated once again during both walks tonight. Americas starting around 3am CEST (again pretty much exactly with the first SoJ sale) and then again during the Europe walk around 7:15 CEST. I for one wasn't able to successfully open a single page for...well, I gave up until now actually and API kept failing (30s timeout) for 15+ minutes during the first, at least 6+ during the second.

Not complaining, just pointing out times/occurrences in the hopes that it may somehow help you address bits and pieces moving forward.
7
User avatar

atari 429

Europe XLinux
During the last days performance was often bad for me. A few times I only got error pages or it took pretty long to load a page. Hope can you find and fix the cause.

Image
7
OP
User avatar

Teebling 5763Admin

Europe PC
I have temporarily disabled the 'Online' filter in the browse trades area - I believe it has something to do with the process hanging/bottlenecking as it was left-joining the sessions table (which grows quickly to 300k entries in half a day, sometimes faster than that).

So yeah, temporarily disabled that, cleared sessions, cleared cache, and a full rack restart as well. I also added some extra cacheing to member profile pages, so some stats on there may take 5 minutes to update after any changes.

Want to observe if things get better following this change. If things do get better and I don't have to keep issuing restarts, can deduce whether the online filter in browse trades was indeed causing the issues. If that's the case I'll need to look at it again and rewrite it to be more performant, I already know how I will do this.

Wish the server luck guys :P

7
User avatar

Schnorki 3024Moderator

PC
Best of luck, Mr. Server, sir! :p
7

 Deleted User 632 0

 Guest
it was pretty good one hour ago. but now it's taking 15-20 seconds to open this forum page.
edit: one min later it's running quick again.
7
User avatar

Th3ory 385

Paladin Americas PC
Similar experience - sometimes it hangs for a minute or so and eventually loads; inconsistent.

Image
Image
7
OP
User avatar

Teebling 5763Admin

Europe PC
Okay, just disabled an old extension for switching accounts that was firing on every single page load for every single user. I'd forgotten that it was re-enabled for testing. Have restarted again, let's see if that helps.

Edit: Here's what tailing the mysql slow log looks like. It only logs queries over like 0.001 seconds or something like that, so just imagine what it is actually having to cope with :P

Image

7

 Deleted User 632 0

 Guest
so this is what server cancer looks like.
it's working good so far.
7
User avatar

Zelym 123

Paladin Europe PC
It has been running a lot better for the past hour.
This is by far my favorite trading site, thanks a lot for your hard work.
7
User avatar

Th3ory 385

Paladin Americas PC
+1 to Udyret and Zelym - running better as of last 30 mins.

Image
Image
7
Running really poorly again, lost of 524/522 errors for the past few hours, same as last night. Need a bigger SQL instance?
7

 Deleted User 632 0

 Guest
yea. it's running bad for me also. 30 seconds to several minutes to load now.
7
Slowness and various server (5xx) errors here as well.
7
It's quite slow for me too, takes a long time to load or it just hits a server error.
7
OP
User avatar

Teebling 5763Admin

Europe PC
Coming to my wit's end here guys, worked fine before I went to sleep, obviously nothing has changed since the modifications I made yesterday.

It seems that as the sessions table grows and grows the performance also decreases proportionally to it. Whenever I purge that sessions table, the site snaps back into life again. But then as time goes on and that table gets populated again, the site begins to slow, coming to a complete stop about 4 hours later.

So only thing I can draw from that is something in my code is like, querying way too many rows from that table. Or, the table is filling up too quickly.

Sessions table is how the site tracks user timestamps, and therefore things like online indicators, last visits (and therefore read/unread indicators), and a bunch of other stuff work.

I need to review all my commits (and hotfixes) after v1.36 and see what might be causing that to happen. No small feat, we're talking thousands of lines or code here. I'll continue to chop out some stuff later today and see how things play out.

7
User avatar

Schnorki 3024Moderator

PC
Not to make dumb suggestions that you've already thought of anyways but just from reading that, it kinda got me thinking "so what's a relatively new feature that's related to the online/offline bits?".

Other than the online search you mentioned you already tried, isn't the "seller/author online/offline" indicator light in threads also relatively new and hence a potential cause? Just a thought.
7
OP
User avatar

Teebling 5763Admin

Europe PC
Update:

Before I decided to start looking at my code again and chopping things off then observing for change, I thought I'd take a look at Cloudflare analytics again.
Spike from Chinese ASN, this is 7 days of records
Spike from Chinese ASN, this is 7 days of records
Way more requests from the top two ASNs, definitely not within fair usage I specified
Way more requests from the top two ASNs, definitely not within fair usage I specified
Surprise surprise, some jackass from China is hitting the dclone API and the dclone tracker page with huge amounts of automated traffic. Most likely a page refresher or scraper of some sort. I think this correlates with the time that things started getting (much) worse?

I set a firewall rule to block traffic from these two networks. Within 1 minute it had already blocked 700+ requests. Within 10 minutes it had blocked 10,000 requests:
Firewall rule working to block malicious traffic
Firewall rule working to block malicious traffic
10k x 6 = 60k requests per hour
60k x6 = 360k requests per 6 hours
360k has been the typical row count of the sessions table before I've had to purge it (usually 6 hours after the last purge), maybe this is the issue, or a strong component of the issue?

Hoping that this makes a difference and will continue observing analytics before I start looking at potential internal issues again.

Teeb

9

Advertisment

Hide ads
999

Greetings stranger!

You don't appear to be logged in...

No matches
diablo2.io is supported by ads
 

 

 

 

Value:
Hide ads forever by supporting the site with a donation.

Greetings adblocker...

Warriv asks that you consider disabling your adblocker when using diablo2.io

Ad revenue helps keep the servers going and supports me, the site's creator :)

A one-time donation hides all ads, forever:
Make a donation