Common issues/problems (what is being worked on for the next website patch)

This topic is now outdated. Please read the patch notes for v1.0.

Hey, admin here.

diablo2.io was a very small, underground community only 3 weeks ago, and the trade section was something I quickly hobbled together in one week before launch as a side-kick feature for the 50 or so active users here and nothing more. I didn't expect it to become so popular - we didn't have an audience here who expected robust trading features. The site didn't have 20,000 members registered. It wasn't designed to be used by 2000+ concurrent online users at any time. Usage has shot up by the multiple-thousands of percent and it's been a struggle to even keep the site online since then.

A lot of bug reports/feedback are coming in from newer members who don't realise that this place is new, who don't realise that the trade section was never designed to be used on this scale, and most annoyingly for me personally - don't read what other people are reporting before they post themselves. It's usually the same things said over and over again, and to save having to repeat myself in every single thread, someone suggested I make a 'common issues' post and sticky it in the hope that people see it before they go ahead and post about filters not working correctly for the 20th time that week for example.

So here are some of the things that WE KNOW are not working and are being worked on RIGHT NOW for the next update.

Speed

This isn't a static site serving blog content, it is a living, breathing thing that has to process thousands of requests every second for all the trading activity, user profiles, notifications, database updates, and forum discussion, to name just a few of the myriad of things happening back end. The site wasn't designed to be used by this huge number of people, and therefore many of the queries I wrote (last year!) are simply not optimised and take a large amount of server processing power to handle at this kind of level.

Query optimisation, cacheing, and further performance improvements have already been made since the traffic spike as emergency measures to stop it from going dark. I have upgraded the server 3 times already to cope with the current traffic and level of optimisation. Performance continues to improve as the site is being continually re-engineered to handle these new and unprecedented levels of users. I monitor the traffic and manually tweak the server config throughout my working day and do everything I can to keep it usable during these times.

I know that the site is slow at peak times, probably better than all of you, because I spend all day on it. If you feel frustrated just imagine how I feel.

What is being done about it? Much of the database query logic is being rewritten in a more performant manner for the next patch, and to aid me in managing the server and discussing optimisation I have the help of two experienced developers. This is a seriously big job but it will ensure that the site remains in the long-term a stable and fast way for you guys to trade.

Filtering for Hardcore, other platforms etc. not working properly

The first implementation of the trade system assumed that about 3-4 trades would be posted a day. This allowed me to load the entire marketplace of a hundred or so active trades on one page, and then filter them client-side with the Isotope.js library, as it is done on the other parts of the site (see the different database sections for example). So the filters were working just fine back then.

However, when I logged on and found 40,000+ trades all of a sudden, that became untenable - if I tried to render 40,000 trades on your screen at the same time not only would my server just shut down but probably your browser would hit its Memory limit and crash, forcing a reload or even worse a lock-up of your own device. Not to mention how long it would take the JS to filter through all of the results on the page.

I know full well that it is ridiculous to have to navigate from page to page and apply a filter every single time. People click a filter for Hardcore and see no results - because currently the pages are only showing the 50 latest trades, and if one of them isn't a hardcore trade for example, nothing shows up, and this makes people mad.

What is being done about it? - I am doing a complete overhaul of the logic behind filtering in the upcoming patch 1.0, it will allow you to filter ALL trades by platform, region, hardcore/softcore, ladder/non-ladder etc. in a paginated way that is server-friendly with having 2000+ constantly refreshing it. In the meantime, use the market search on the /trade page and look for the icon that indicated hardcore/playstation etc. to find trades suitable to you.

How do I delist/delete a trade that I no longer want on the market?

At the moment you can currently only do this if the trade has received no offers. Just click the little 'X' on the trade description (the first post by you) and it will delete the trade. However, if it has offers, it cannot be deleted by yourself - you would have to contact

Stormlash or Beardozer (moderators) and ask them to do it for you. Alternatively, just wait 3 days for it to auto-expire. If the trade receives no comments or offers during this 3 day period, it will delete itself automatically.

What is being done about it? - I am having to hack some of phpBB's permissions system to allow you guys to sort of have moderator permissions but sort of not have them, so that you will have the ability to delist a trade completely with no hindrances. Coming in the next patch.

I can't mark my item as sold

You CAN mark your item as sold - you're probably just pressing the wrong button. This is because most are scrolling to the bottom and pressing 'Submit' after having entered the details of the sale. Do not press Submit, press the 'Mark as Sold' button immediately after those fields. Then it will work.

What is being done about it? - Obviously this is bad UI and people are getting confused, and it doesn't occur to them that the Submit button doesn't do anything in this case. So once again, this is being changed in patch 1.0, which is being worked on all the time.

I can't give trust as a seller to my buyer

Currently you can only give trust as the buyer of the trade - that is, you are only eligible to give trust if you have been named as the purchaser of the trade, and then see the 'give trust' button on the trade. Also, there is supposed to be a notification for when an item is marked as sold - but it is currently out of action because a rogue query in this part of the code was causing thousands of emails to be sent a second and crashing the server. This is changing in the next patch.

What is being done about it? - Next patch you will be able to give trust both ways (both buyer and seller can 'vouch' for each other), and notifications will be there when an item is marked sold, for anyone watching the topic.

My trade isn't showing in search/profile/latest trades/on the item entry

I have had to introduce a cache for some of the queries associated with trade creation. If you don't see your trade immediately on the various parts of the site, it is because another query hasn't yet been made and it is still showing cached content. Just wait a few minutes and it will appear eventually. This was an emergency measure put in place to ensure that the site can continue to function at peak times.

What is being done about it? - Due to the sheer volume of trades coming through on the site, this will likely remain in place into the future or until traffic reduces. It may be the case that optimisations in other places allow the site to relieve this cacheing requirement. Not clear yet whether that is possible.

And finally some other questions that are getting asked non-stop that can be answered here and now:

• Are you going to make it easier to search for, filter for, and create trades for magic and rare items such as charms? Yes
• Are you introducing WTS/WTB tags, and additional trade creation options like Ethereal/no. sockets etc? Yes
• Are you introducing an advanced keyword search allowing searches of offers within trades as well? Yes
• Will I be able to paste clipped screenshots straight into the editor? Yes
• Will I be able to make trades for different platforms and regions rather than just base them off my account settings? Yes
• Are tooltips and autolinked items in text going to return as they were before the tidal wave of users? Yes
• Are you starting to lose your mind having to answer the same complaints every single day? Yes
• Why is the site so slow? Seriously, again? Scroll up if you somehow reached this sentence without a detailed explanation from me somewhere.
• When will this update to v1.0 be ready? When it's ready
• Why don't you get some help from other people instead of doing everything by yourself? I have help from other people already
• Do you need help from me? I am a developer etc. - No, I don't, everything that can be done is being done

Thanks. Now that I have taken the time to make this summary, and posted this and stickied it in a prominent place, I'm hoping to not have to waste precious time responding to the same questions repeatedly, and will have some time to actually work on fixing the issues. God help the author of the next 'why is the site slow' post that will inevitably pop up. If you guys see stuff like this in bug reports/feedback, please just post a link to this thread and ask them to actually look around before they post.

Doing everything I can to bring you guys a site that meets your expectations. Rome wasn't built in a day.

Was going to post something about the current issues but hey! You answered all my current questions. This is really a promising site, and would love for this site to grow more and maybe be in our phones someday. Keep it up @Teebling

The sites' quite nice to use for general referencing, put forward in a clean and pleasing to consume manner. Just another reason to keep it open in the background almost always just incase - outside of using it for other reasons/forum etc.

Looking forward to a time when this is made even better by improvements to come.

larlarghost wrote: 3 years ago

maybe be in our phones someday

The site is fully responsive and anything you can do on desktop you can do on mobile.

Good productivity notes \o/

I look forward to seeing all the changes

Thanks for making this post! As an experienced dev myself I understand what you're going through and don't envy you at all. I'm glad you got some help too

The last thing you might want right now is someone else's take on it, but I only offer this in case it might help you. One thing that could be done to drastically reduce load is having a base page that is full page cached and doesn't include any logged in user type stuff. This is what really big sites often do. Then they ajax in the user specific parts of the page. Many things on this site could be full page cached. I haven't used phpbb in like, 15+ years tho so I don't know what it's like now or how hard that is. My guess is, it's a lot of work. If I was going to suggest an optimal strat though that'd be where'd I'd go for. This means you're serving static pages and then ajaxing in the stuff that matters to the user. A lot of pages this means only the top right corner changes.

It's not without it's things to tackle either. There's some other things too, like getting post times to update via javascript, and clearing the cache of only pages that need it when updates happen.

The benefit of this approach is anonymous users and logged in users get served the exact same full page cache, it can even just be stored as an html document with some caching systems which means zero database and php usage only hard drive bandwidth is a concern. Some things like the random items, can run via javascript too. Like just load in a bunch of stuff to a JS array, and when the cache is invalidated swap out the array for another random one. Then it's still random but also doesn't load every item to the page. Just a few examples off the top of my head.

For the full page cache approach you can see a youtube page do this a lot of the content isn't there and then is loaded in. This makes most of the php/mysql never even touched on the majority of requests. Again I don't know how much work this is to achieve in phpbb, my guess is way too much. Take the home page for example all of it but the upper right user section could just be a static page generated every so often when cache expires and the rest of the time it serves a stored static html document, or something in a full page cache Memory like varnish, or nginx full page cache, or litespeed cache. But this can even be extended to searching and ajax in some cases... but I've ranted long enough and prob wore out my welcome lol

I have to say you've done an amazing job to handle so much traffic and keep the site going. I think most sites would've failed by now it's a testament to your hard work so I think you should feel proud of that Thanks for your hardwork. When I saw you were spending 12 hours a day on it that's when I sent a donation you deserve it.

Thanks @Fae.

Fae wrote: 3 years ago

doesn't include any logged in user type stuff

The online stats is only for the last minute, so it's not a slow query any more. It was slow when it had a 10 minute span, but not now.

Fae wrote: 3 years ago

Many things on this site could be full page cached.

Most things are already cached - dynamic stuff (trades) for a couple of minutes and static stuff (database entries) for a month. phpBB achieves this quite easily with its DBAL functions.

Whether that's on server-side or client-side, there's already a lot of cacheing happening. Furthermore the site utilises Cloudflare's CDN.

Noted on using JS to swap out arrays etc. - seems a lot of work on top of what we've already done (and are continuing to do) though - so might leave that one on the long term optimisation list.

Cloudflare CDN can full page cache but it lets through a lot of requests and iirc if it detects cookies it often doesn't do anything at least as far as the full page cache is concerned (it still does a lot tbh even then but talking FPC here).

DBAL is database abstraction layer? That means it's already hitting php at the very least.

There are a lot of diff types of caching, and most php stacks have multiple layers of caching built in. Opcode, browser, mysql has query caching if it's enabled (tho it shouldn't always be), and so on.

Just like there's multiple levels of caching and types, not all caching is built equally. Full page cache is the holy grail of caching basically because it stores all the output from all the php/database calls into a static full page in Memory or a file (which is then loaded from Memory). On a site like this though, there's things that constantly change as you said. Users online, post time, and anything user related if they're logged in. So the goal of what I said is to full page cache the entire page, all the the time, as an anon user would see it. Then ajax in on top if it user related stuff or stuff that changes, so that the only queries/php you run are ones that are necessary. This drastically cuts down server load because you're running a very small fraction of the stuff you ran before, for the majority of requests.

Again I'm not sure how much work this is for modern phpbb, but if you wanted to be able to run this site on an inexpensive server that's how I'd go about it. It's basically re-engineering how phpbb outputs at that point tho the nice thing about it is the surface area of things to optimize also is drastically reduced. Only things you ajax request become hit often.

If the advice was unwelcome that's fine it's not my site, just wanted to offer some of my perspective if it was welcomed. Wish you the best either way

Edit: one example is the user profile page. You could cache literally all of it, then ajax in the upper right bell and user menu. And you're done. These full page caching systems if smart, also let you have the best of both worlds as it has a way to flag pages that would reflect changes then on the next request to that page it generates a new full page cache but that's far more complicated and tbh most the stuff here could live with waiting 5 minutes or more for an updated page, but the end goal would be something that both updates and is full page cached as much as possible. Anyways cheers

TY for post. Donated today, cant wait for server improvements!

Oh man, you're not only a talented developer but also a stakeholder management wizard. Next donation is on its way, keep up the awesome work, glad you got help!

Your my hero.

Thanks for working so hard on making this site work for something it wasn't originally intended for! The improvements so far have definitely been noticeable.

I've been trying to keep tabs on everything coming next patch & every time I read something I get more excited for it! Seeing it laid out like this just boosting my excitement even more, but I will try to contain myself.

Should've put "if its too fast it won't feel nostalgic" at the end of Speed .

But ive noticed huge improvements to loading times recently and its been working great for me whenever im on it.

You sir, are a hero. If only Blizzard could work this hard on keeping the game up =)

Awesome! I thought it was me that was seeing the site slow.

However I haven't quit and gone to another site. It's a very nifty site.

Saw some good answers from members, already traded items from good members.

It's been a good experience so far!

Truly appreciate the effort that was put in this website. Thank you

Patience is Virtue.

Key point is that the initial Trading function was deployed to see how the market would react and at its time had 400+ members now up to 20K+ to your point @Teebling . Understand that the market has irritable tendencies, but such is the D2 Community in general. Stay the course buddy. Folks obviously see the value else they wouldn't be managing up the feedback. Have the Faith in you and the brethren to continue to optimize and evolve the site. I think everyone needs to take a step back and realize that this truly was a platform launched as a Passion project by an individual practitioner and has easily ascended quicker than majority of other notable and equitable platforms in D2. Let us not forget, that this is also the main trading platform outside of D2JSP so folks need to roll with the punches and remain committed. Use in-game, use Discord as you should, but you can also continue to lean into D2IO as it evolves. Cheers All.

Teebling wrote: 3 years ago

I am having to hack some of phpBB's permissions system to allow you guys to sort of have moderator permissions but sort of not have them

Sweet jesus, please test this before it goes live

Hey, just wanted to say I think you've built something really incredible here, and you should be proud of yourself

All I can say is thank you, you have made a wonderful place here. I googled looking for a site to trade that WASN'T D2jsp for D2R and a link to the website here was one of first ones I saw. And here I am, and loving the trading features so very much! So I can tell you that you were high on the google search and that probably is helping bring more eyes, even if that is more server fires. Thank you, so very much. Keep up the great work, I will be here to trade so very much. You are amazing <3