Screenshots of drops

there are some easy to handle tools for it in python, i'll try them maybe next week
can let you know about my progress, but the online ocr readers work quite well with the new screenshots already!

Sabcoll wrote: 4 years ago

there are some easy to handle tools for it in python, i'll try them maybe next week
can let you know about my progress, but the online ocr readers work quite well with the new screenshots already!

If it were to be integrated into screenshots for Trade here on the site etc. it'd have to be PHP with a jQuery frontend that can hook into a library called PLupload.

Then you might give tessaract a shot.
It's what I've been using in python..
https://github.com/naptha/tesseract.js#tesseractjs

Sabcoll wrote: 4 years ago

Then you might give tessaract a shot.
It's what I've been using in python..
https://github.com/naptha/tesseract.js#tesseractjs

Interesting... will have a look at this for next patch! Would be cool to dynamically change the stats on items in trade ads and the holy grail tool.

Yeah,
that's what I'm trying to achieve.
I want to screenshot my items (will be saved in D2 Screenshot folder) and have my python script run over the Screenshots and analyse the items within.
Then write a file with the included text so I can easily post them into forums (for trading)...

Okay, so after only a few hours of messing around, I came to the conclusion that I would need to start at the very beginning:
Creating a D2R Font File and train a whole tesseract model to "read" the Font File Characters.

I guess this will take a whole lot longer...

Sabcoll wrote: 4 years ago

Okay, so after only a few hours of messing around, I came to the conclusion that I would need to start at the very beginning:
Creating a D2R Font File and train a whole tesseract model to "read" the Font File Characters.

I guess this will take a whole lot longer...

:O Sounds nuts dude, do you think you're still going to continue?

Yeah, I will start with the font file today and hopefully be able to create some training data for tesseract already.

i also posted on reddit if somebody has already checked the MPQ Files for the font (as it was done with D2 in the past)

Cool idea @Sabcoll

The default font for D2R items was Exocetblizzardot-medium.otf

I extracted it from casc data storage using CascView, it was in `D2R\Data\hd\ui\fonts`

Here is the otf file https://easyupload.io/l17468

@thrugg_aka_trogg wow, thank you very much.
I have been training with exocet-blizzard-light.ttf but I'm not too happy with the outcome.

I attached a few examples.
Especially ":" and "+" gets heavily confused.
I guess i need to apply some filters before doing the actual OCR Reading.
See my results here:
https://imgur.com/a/icOjUBB

will try to get some better results with the new font!

@Sabcoll really impressive so far though, well done!

Im not too happy with the outcome, but its the first time for me trying to get my own ocr font working.

I guess a good idea would be to create a dictionary and implement it, instead of relying on the character recognition only.

Will be an interesting topic to work on.

Maybe it would be a good way to create a holy grail tracker or something similar.

Sabcoll wrote: 4 years ago

Maybe it would be a good way to create a holy grail tracker or something similar.

Would definitely be a sick way to input the items yeah. Just drag and drop a screenshot and added to grail.

This is my current state of "learning" the D2 Font.
https://imgur.com/a/ExV2g9h

I'm pretty happy so far with the outcome and that without using any filters yet.
But a little bit of postprocessing done by tesseract already.

If I can manage to write some rules, e.g.
(41 -> If there is a "(" there has to be a ")" too.
So 1 has a lilelyhood and will be replaced by ")".

Or ' followed by a number is not allowed, so ' has to be a + ...

I need to think about the effort to do that or if it's just easier to correct it manually..

Sabcoll wrote: 4 years ago

This is my current state of "learning" the D2 Font.
https://imgur.com/a/ExV2g9h

I'm pretty happy so far with the outcome and that without using any filters yet.
But a little bit of postprocessing done by tesseract already.

If I can manage to write some rules, e.g.
(41 -> If there is a "(" there has to be a ")" too.
So 1 has a lilelyhood and will be replaced by ")".

Or ' followed by a number is not allowed, so ' has to be a + ...

I need to think about the effort to do that or if it's just easier to correct it manually..

That's pretty cool, nice work!

This got me curious and decided to challenge Google lens and I must say that I'm impressed.

The following image was used And this is the output of 2nd try

Que-Hegan's Wisdom

Mage Plate

DEFENSE: 644

DURABILITY: 33 OF 60
REQUIRED STRENGTH: 55
REQUIRED LEVEL: 51

+1 TO ALL SKILLS

+20% FASTER CAST RATE

+20% FASTER HIT RECOVERY

+146% ENHANCED DEFENSE

+15 TO ENERGY

MAGIC DAMAGE REDUCED BY 6
+3 TO MANA AFTER EACH KILL

HOLD SHIFT TO COMPARE

This is just one image tried and sometimes the carriage returns aren't working with some data ending up on the same line like below.

Que-Hegan's Wisdom

Mage Plate

DEFENSE: 644

DURABILITY: 33 OF 60 REQUIRED STRENGTH: 55 REQUIRED LEVEL: 51

+1 TO ALL SKILLS

+20% FASTER CAST RATE

+20% FASTER HIT RECOVERY

+146% ENHANCED DEFENSE

+15 TO ENERGY

MAGIC DAMAGE REDUCED BY 6 +3 TO MANA AFTER EACH KILL

HOLD SHIFT TO COMPARE

Overall impressed by the accuracy though.

Banking on your solution tho with a trained model with the actual font.
Keep up the good work.

@mengstrom woah, that's amazing!
There are so many possibilities out there, thanks for mentioning google lens.

There is also an API for it:
https://codelabs.developers.google.com/ ... i-python#0

I took some screenshots from MrLlamas Stream.
Because I took the pictures from his Stream and he has the tooltips over his inventory (see all the charms) tesseract has some trouble with the font.
Your picture of Que-Hegan's is "pretty perfect" to be used, but it's not the reality.

Can you maybe check the google lens with the pictures from MrLlama?
See my links, on the left is the "original"...

Sabcoll wrote: 4 years ago

Your picture of Que-Hegan's is "pretty perfect" to be used, but it's not the reality.

Just chiming in to say that many will be using things like gyazo or other web services for screenshots/uploads. These will definitely compress the images, introducing distortion and artefacts, that I'm positive will affect the performance of any OCR stuff.

Sabcoll wrote: 4 years ago

@mengstrom woah, that's amazing!
There are so many possibilities out there, thanks for mentioning google lens.

There is also an API for it:
https://codelabs.developers.google.com/ ... i-python#0

I took some screenshots from MrLlamas Stream.
Because I took the pictures from his Stream and he has the tooltips over his inventory (see all the charms) tesseract has some trouble with the font.
Your picture of Que-Hegan's is "pretty perfect" to be used, but it's not the reality.

Can you maybe check the google lens with the pictures from MrLlama?
See my links, on the left is the "original"...

So I tried with you screenshot and this is the result. Note that this is a download of your posted file which isn't the best resolution since it's snipes from his steam and then compress by imgur afaik.

With an ingame capture I think the result would be much better.

EDGE

TIRTALAMN

Twe-HAND DAMAGE: 4 TO 8

REQUIRED DEXTERITY: 35

REQUIRED STRENGTH: 25

REQUIRED LEVEL: 25

Bew CLASS VERY FAST ATTACK SPEED

LEVEL 15 THORNS AURA WHEN EQUIPPED

+35% INCREASED ATTACK SPEED

+361% DAMAGE TE DEMONS

+280% DAMAGE TO UNDEAD

+75 PRISON DAMAGE OVER 5 SECONDS

7% LIFE STOLEN PER HIT 52


My idea for the use of OCR would be to capture items ingame with the use of AHK while I'm playing.
Press a button and draw a capture region around the item tooltip as to minimize all the noise from the screen, perhaps even decrease the opacity of the Black background of the tooltip to increase the chance of a better result.
Hook it up to @Sabcolls tool or google lens to extract the text. This way, the screen shots would be near perfect everytime so chances of success would be pretty high.

My personal use-case would only need to capture the item name and hook it up to some holy grail tracker.
Add the entry by name if not exista and include the screenshot. That's it.
Added bonus would of course would be to capture the item tooltip entirely for other purposes. But for me playing SP, that's just a bonus.

If anyone is interested in doing some webdevelopment on the topic,

there are already pretty good tools from tesseract.js

https://www.npmjs.com/package/tesseract.js/v/2.1.1

Sabcoll wrote: 4 years ago

If anyone is interested in doing some webdevelopment on the topic,

there are already pretty good tools from tesseract.js

https://www.npmjs.com/package/tesseract.js/v/2.1.1

Built a little quick and dirty test React app Saturday that uses the PLupload library Teebling mentioned to load the images locally, display a preview and run it though tesseract.js using the default eng language data. I'm still working on making my own training data when I have time (never messed around with that so it's a learning experience). I'm hoping the text recognition speed increases with better test data, as it is now with eng it takes several seconds per image.

Was also thinking as far as the accuracy goes, I think the most important part is just getting the numbers correct. The rest can be error corrected by comparing the text against a dictionary of item attributes. If the item name can be properly identified, the dictionary can also be reduced to just the subset of possible attributes for that item which would hopefully eliminate false matches. Could use a dictionary lookup on the item name as well against just a dictionary of possible item names.

Spoiler

Reminder this is without training data for the d2 font. Guess you have to click to view he gif. Also the recording is glitchy, loading the previews is quicker, first OCR result was longer :shrug: