Awesome service!

Leading up to D2R launch I've been looking around for a good resource to look up items.etc and most of the resources out there imo weren't very good. This is exactly the kind of thing I was looking for so thanks for providing this service for the community!

I just had a quick question. Is there perhaps any plans to provide an API to search the database? I would love to incorporate this into a project I've been working on. It doesn't appear to be possible to search via URL parameters and I didn't notice any publicly posted API's so just thought I would ask. If not no problem!

jcr4990 wrote: 3 years ago

Leading up to D2R launch I've been looking around for a good resource to look up items.etc and most of the resources out there imo weren't very good. This is exactly the kind of thing I was looking for so thanks for providing this service for the community!

I just had a quick question. Is there perhaps any plans to provide an API to search the database? I would love to incorporate this into a project I've been working on. It doesn't appear to be possible to search via URL parameters and I didn't notice any publicly posted API's so just thought I would ask. If not no problem!

I love this place also . I'm sure someone will answer your API question.

Hey jcr, you're welcome!

I've had a bunch of people asking about a public API - things I just don't know anything about it! It's a standard MySQL database here, how that hooks up with an API I'm not sure. If you or anyone else tech savvy could give me a quick overview of what would be required that might help me getting it off the Ground.

@Teebling
Not sure how this is done in php but at its core, an API is a interface for your service ( diablo2.io) that let others (be it another application or user) query data you expose.

Let's imagine one would want to fetch all unique items from the database.
Expose an endpoint (i.e. diablo2.io/api/items) that takes in a object specifying the request to make.

{
"type": "unique"
}

Then on you backend (guessing a controller/page) handling said request, parsing query to figure out what type of query to do against the database.

There's more than one way to skin the cat and this is just a hint to get you started.
Recommendations would be to implement rate-limit and authentication (api

Key) or other means of authorizing the caller.
But most importantly, make sure to parameterize queries to the database as to not expose to sql injections. Sanatizing and validating incoming data is a good start as well to keep yourself safe

And please keep in mind that this puts a lot of stress and traffic onto the SQL Server and might cause trouble with your hoster...

Teebling wrote: 3 years ago

Hey jcr, you're welcome!

I've had a bunch of people asking about a public API - things I just don't know anything about it! It's a standard MySQL database here, how that hooks up with an API I'm not sure. If you or anyone else tech savvy could give me a quick overview of what would be required that might help me getting it off the Ground.

Mengstrom's post above explained it much better than I could have. Unfortunately I'm no expert developer. I could whip up a basic API in Flask but I'd have no idea how to do authentication or rate limiting or protection against sql injection attacks. I just wasn't sure if this was an already planned feature or something you had considered or not. Obviously there's no pressure I was just curious!

Was pondering over this again today and thinking that I'm not really sure an API would be of much use tbh.

Diablo 2 Resurrected stores a lot of data in JSON format - something people weren't expecting but it's certainly made it a lot easier for developers to gain access to data.

If I wanted to build a tool, why would I want d2io's data through an API when I can take it from a bank of JSON documents just as easily?

Just a thought.

already stores items in json files on disk?

If those files (and their content) is as easy to search in as on your site, then no, there wouldn't be much of a point.
However my guess is that you've extrapolated much of the information available and connected the dots with features like showing their relation to other items like tunes within a runeword.

Basically the search and filter on the site as everything already. What's left is to extraxt that functionality into an API. An API that both the site and external users could leverage. No need to double up on the code side, just point the backend of the search to make use of the api.

I guess the main thing here would be the heavy lifting already done by you and this community.

Teebling wrote: 3 years ago

Was pondering over this again today and thinking that I'm not really sure an API would be of much use tbh.

Diablo 2 Resurrected stores a lot of data in JSON format - something people weren't expecting but it's certainly made it a lot easier for developers to gain access to data.

If I wanted to build a tool, why would I want d2io's data through an API when I can take it from a bank of JSON documents just as easily?

Just a thought.

Perhaps an explanation of what I intended to use an API for would help. I have a small guild discord of friends that have played various games together for years. There's a handful of us intending to play D2R at launch. I've done a fair amount of discord bot coding for various things over the past couple years and I had a couple people requesting a discord bot command to look up D2R items/runes.etc for our guild server. My first idea was to just scrape data off the d2 wiki but as I started playing around with it I almost immediately noticed that there's very little uniformity in how items and item data are displayed on the wiki. Most item pages have a <table> html element so I started out pulling all the table data and attempting to format it so it would display nicely in Discord. Before long I noticed some items have no table element at all and the item data I was looking for isn't very easy to parse out on certain pages without resorting to less than ideal methods like regex. It got very messy very fast and I decided to abandon that idea and look for alternatives. That's basically how I stumbled across diablo2.io.

I noticed right away that all the item info I needed was uniformly displayed on every page and it's all very clean and nice looking. My hope was that there would either be A: a public API I could use to fetch item data in json or some other format OR B: a /search/ route with parameters to directly pull up item pages without using the search drop down menu. Unfortunately it seems neither of those are currently an option. I also don't know if you as the developer/owner of this page would be okay with people doing light scraping for small personal projects or not. As it stands I still like this site a lot and will definitely continue to use it but may need to continue my search for another source of data for my project.


Edit: Upon re-reading your post it sounds like you're saying D2R stores its item data locally in JSON format? If that's true I was completely unaware of it and to be honest I'm not quite sure how I would access it. At a glance all the data files in my d2r beta directory don't appear to be in json or plain text format that you can view with just any normal text editor. This is a new thing for me!

The game data is stored in container format called CASC that bundles all the files together. You can get a lot of data from the json files within, but unless I've missed something, the attributes of items are stored in binary files that would have to be properly decoded to be usable.

Idk if Teebling is ok with people using the search endpoint externally, but it has a pretty minimal html result, should be easy enough for that to be a working solution

If you are ok doing some (a lot) data processing work, there's a few repos with the D2 data:

https://github.com/mattinm/DBD2 : Text files that are actually csv's with D2 game data
https://github.com/blizzhackers/d2data : JSON docs with D2 data as well

I haven't fully looked through it all yet to know it has everything. I think the csv's only have attribute

Key names, not the actual labels, so that data would have to be sourced elsewhere or manually gathered. (Edit: Actually it looks like in the d2data repo, there is a Properties.json that has all the labels)

All the d2r image assets can also be pulled from the beta client using CASC viewer (http://www.zezula.net/en/casc/main.html) to extract it and a python script (https://github.com/dzik87/d2r-sprites/b ... sprites.py) someone made to convert the images to pngs.

All that being said I can see the use case for an API (even if a JSON data dump of all d2 data and the assets were easily accessible, which isn't exactly the case) as there are still applications like jcr describes, where you have a distributed client (discord) that needs to be able to pull the data from somewhere externally. Having an API to consume for this would save people the time and energy of building their own. It could draw also traffic to d2.

Io if you make that part of the license agreement to use the API to require a backlink. If you require a API

Key as well you can monitor usage and block access should it be abused.

Noemard wrote: 3 years ago

Idk if Teebling is ok with people using the search endpoint externally, but it has a pretty minimal html result, should be easy enough for that to be a working solution

I can add a simples /search/ type thing for use of bots etc. but this will have to be post-release guys, there's already so many feature requests that I've got to work my way through before I can reach that.

Noted on all the points raised in this discussion by the way, thanks for all your input guys.

Teebling wrote: 3 years ago

Noemard wrote: 3 years ago

Idk if Teebling is ok with people using the search endpoint externally, but it has a pretty minimal html result, should be easy enough for that to be a working solution

I can add a simples /search/ type thing for use of bots etc. but this will have to be post-release guys, there's already so many feature requests that I've got to work my way through before I can reach that.

Oh, sorry, for that comment I wasn't suggesting you update anything. I just meant I didn't know if you were ok with jcr consuming the search from a discord bot.

It's simple enough to parse the html response from search to get the item name & id. Then make the request to get the tooltip and link to the item page.

Teebling wrote: 3 years ago

Hey jcr, you're welcome!

I've had a bunch of people asking about a public API - things I just don't know anything about it! It's a standard MySQL database here, how that hooks up with an API I'm not sure. If you or anyone else tech savvy could give me a quick overview of what would be required that might help me getting it off the Ground.

I work for a very large gaming company doing public-facing API work and API security and I would be willing to help out if you need.

For those familiar with Docker & GraphQL, here's a great API solution for querying D2R data:

https://github.com/micheljung/d2r-api