Slowness today

Alright so I guess the consensus is that there has been an improvement but it's still not quite as snappy as people would expect.
Some 503 timeouts are to be expected and within the realm of acceptability for me, but long page loads are not.

So if the latter is performing okay, and I haven't had to restart for 16 hours, I think the two issues responsible were:

• Online trades filtering in both browse and search tools, which I will need to rewrite in a more performant way next patch. I think that the queries were so heavy that they just hung forever, taking up processes and failing to die after timeout threshold.
• And the spike in automated traffic to the tracker page, which I have now firewalled. It has blocked nearly 1 million requests since yesterday - certain that this was also a contributing factor to the slowdown in general.

I need to take a look at trade listings again, and stash tabs, and perhaps leverage cacheing a little more heavily there too. Couple of other things on my mind that might help, but yeah, glad it's been at least accessible since the last changes. Progress!

Progress indeed!
I even got to report a 5/6 during the walk just now (which would've normally been impossible). Though it did go back to not loading anything after that. ^^

Has been running much more smoothly on my end Teebs! Thanks for the detective work.

Just found the site, worked for around 10 minutes. was happy. since 1 hour now... not possible to work with the site.
endless loading.
timeouts.
522s....

Sad. Thought I found a neat alternative to the dying fg-trading...

Aye, been getting worse again

Outage today: traffic spiked to 5k concurrent due to walks, knocked out mysql database server.

When purging sessions table via phpBB ACP, it somehow managed to half-drop and corrupt the sessions table instead of truncating it. I've had to spend the last 40 minutes reconstructing that table and repopulating it.

That's the last time I'm going to trust the admin panel for that task, will do it from command line from now onwards.

ahh now that makes sense I guess. Thanks for the quick explanation and fixing

Do you need money to make the website run better? Just curious. I feel like you should have these issues fixed by now. Maybe tell the community what you need whether its more server people or $$. I'm sure everyone will help you.

Asuna. wrote: 2 years ago

Do you need money to make the website run better? Just curious.

Throwing money at the problem doesn't help - already on a top quality dedicated machine with full cacheing and CDN etc. More infrastructure won't help. Current costs right now around 400 USD a month for hosting alone. The problem is my own code not running performantly, not my hosts or the server itself.

Asuna. wrote: 2 years ago

I feel like you should have these issues fixed by now.

I monitor the site for most of my waking day, have spent years building it, maintain it completely by myself, whilst maintaining IRL duties on top. Please don't add extra expectations on top of what I already have to deal with in terms of the pressure I put on myself to keep the site running.

With multiple thousands of concurrent users spiking in a way that is impossible for me to simulate in a dev environment, new bugs and issues are discovered every time an outage occurs. It's hard to tell when fixes are effective, whether they'll proactively help with the next wave, or whether they do nothing at all.

The best I can do is continue to monitor, make changes when and where I can, and then hope that it works out.

Asuna. wrote: 2 years ago

Do you need money to make the website run better? Just curious. I feel like you should have these issues fixed by now. Maybe tell the community what you need whether its more server people or $$. I'm sure everyone will help you.

There's a donate button that you can use at the navigation bar at the bottom Even if it won't solve all the issues it is still a good way to make Teebling and the dev team know that you appreciate their work.

Charmani wrote: 2 years ago

Just found the site, worked for around 10 minutes. was happy. since 1 hour now... not possible to work with the site.
endless loading.
timeouts.
522s....

Sad. Thought I found a neat alternative to the dying fg-trading...

i'm glad you found the site. I prefer it over FG sites simply because i don't like real money trades, or those that revolve around someone elses real money. If it's your first time give it a chance, this site is amazing, and eventually too much traffic wrecks anything, so let the Devs and Teebling work out the issues and it will get better!

Getting hit hard again right now it seems. :/
(Possibly user spike due to asia progress?)

It happens often with the dclone progress on the servers. That said, even with the slowness, this is still my go-to trading site, thanks to all the work and dedication of the team so far.

Schnorki wrote: 2 years ago

Getting hit hard again right now it seems. :/
(Possibly user spike due to asia progress?)

I was online last week sometime between 4-5am EU CET, d2.

Io went 404 & within not even 2min uber counter went from 1-5.
So yeah, people definetly using ddos to press ani prices.

Teebling wrote: 2 years ago

Amethia83 wrote: 2 years ago

i noticed the diablo

Io site stopped being slow all of a sudden

Narrowing it down still, think I've found the core issue - sessions table filling up to 600k rows overnight with useless guest sessions that just take up processes but don't do anything. Just truncated and things snap back into life.

I will set a mysql event to clear guest sessions every hour from now on, that should keep things snappy throughout the day, at least until I work out why they're being stored in the first place. Will implement later tonight.

Just implemented with hourly interval, might decrease that to 30 mins. Let's see how she sails now.

Teebling wrote: 2 years ago

Narrowing it down still, think I've found the core issue - sessions table filling up to 600k rows overnight with useless guest sessions that just take up processes but don't do anything. Just truncated and things snap back into life.

Could it be deliberatly? It almost feels on purpose

Teebling wrote: 2 years ago

Teebling wrote: 2 years ago

Amethia83 wrote: 2 years ago

i noticed the diablo

Io site stopped being slow all of a sudden

Narrowing it down still, think I've found the core issue - sessions table filling up to 600k rows overnight with useless guest sessions that just take up processes but don't do anything. Just truncated and things snap back into life.

I will set a mysql event to clear guest sessions every hour from now on, that should keep things snappy throughout the day, at least until I work out why they're being stored in the first place. Will implement later tonight.

Just implemented with hourly interval, might decrease that to 30 mins. Let's see how she sails now.

Out of curiosity, what creates a guest session here?
Is that any (unique?) visitor to the site, actual logins, browsing of certain sections?

Schnorki wrote: 2 years ago

Out of curiosity, what creates a guest session here?
Is that any (unique?) visitor to the site, actual logins, browsing of certain sections?

Any visit of any page in which the user's session ID isn't tied to a registered user. Table populates with rows of session_user_id=1. I think phpBB does this because it has an option (which is disabled) to enable read/unread indicators based off of cookies, letting guests track progress basically. To do that it needs to assign that guest a session ID and connect it with the cookie, hence having to populate the rows. So yeah, there is a reason the table gets populated like that, but for my purposes it is unnecessary.

That..does just sound unnecessarily crowding ^^

I think I fixed it, how's it been last 24 hours?

Sessions table was flooding with useless guest entries and then being queried in a really heavy way by a few things, resulting in things locking up. Yesterday, I put in an automatic hourly mysql event to keep that table tidy, and optimised those queries; the site seems to have been fine since then. Haven't had to restart since the change.