help [Need Help] Sorting data from website

[flewis]

Dear all,

 

I have found some data on a website that I'd like to extract. In java I don't know how to get the data from the website and sort it into a file in my scripts folder.

 

For example the website has data like:

"2": {"buy_average": 193, "members": true, "name": "Cannonball", "sp": 5, "overall_average": 195, "sell_average": 194, "id": 2}, "6": {"buy_average": 174492, "members": true, "name": "Cannon base", "sp": 187500, "overall_average": 180888, "sell_average": 176566, "id": 6},

 

Only problem is it's all in one string and I need to separate each item out onto a different line in the text file (the file contains all Runescape items).

 

Website I want to extract data from: https://rsbuddy.com/exchange/summary.json

 

Sorry I'm pretty new to Java so I have no idea what I'm doing tbh xD

 

Thank you for any help though!!

 

-Flewis

[Explv]

Firstly that file will not include all RuneScape items, only ones tradeable on the Grand Exchange.

It is a JSON file, you should use a JSON library to handle the data, such as https://github.com/fangyidong/json-simple or https://github.com/google/gson

Here is what the file looks like prettified:

{
  "2": {
    "id": 2,
    "members": true,
    "name": "Cannonball",
    "sp": 5,
    "overall_average": 195,
    "sell_average": 195,
    "buy_average": 195
  },
  "6": {
    "id": 6,
    "members": true,
    "name": "Cannon base",
    "sp": 187500,
    "overall_average": 179789,
    "sell_average": 177522,
    "buy_average": 176043
  }
}

To extract each item you would create a JSONObject of the whole String.

Then iterate over each of it's keys, these are the item ids e.g. (2, 6).

For each of those keys, get the value (this is the item's information), and then print the value of the "name" key out to a file.

[Explv]

12 minutes ago, flewis said:

Dear all,

 

I have found some data on a website that I'd like to extract. In java I don't know how to get the data from the website and sort it into a file in my scripts folder.

 

For example the website has data like:

"2": {"buy_average": 193, "members": true, "name": "Cannonball", "sp": 5, "overall_average": 195, "sell_average": 194, "id": 2}, "6": {"buy_average": 174492, "members": true, "name": "Cannon base", "sp": 187500, "overall_average": 180888, "sell_average": 176566, "id": 6},

 

Only problem is it's all in one string and I need to separate each item out onto a different line in the text file (the file contains all Runescape items).

 

Website I want to extract data from: https://rsbuddy.com/exchange/summary.json

 

Sorry I'm pretty new to Java so I have no idea what I'm doing tbh xD

 

Thank you for any help though!!

 

-Flewis

 

If this is a just a one off thing though, I wrote a quick python script to do it:

import json

item_data = {}

with open('summary.json', 'r') as summary_file:
    item_data = json.load(summary_file)

item_names = [item_data[item_id]['name'] for item_id in item_data] 
    
with open('item_names.txt', 'w') as item_names_file:
    for item_name in item_names:
        item_names_file.write(item_name + "\n")

 

And here is the output:

https://hastebin.com/dewunamare.sql

Edited July 30, 2017 by Explv

[flewis]

2 minutes ago, Explv said:

Firstly that file will not include all RuneScape items, only ones tradeable on the Grand Exchange.

It is a JSON file, you should use a JSON library to handle the data, such as https://github.com/fangyidong/json-simple or https://github.com/google/gson

Here is what the file looks like prettified:

{
  "2": {
    "id": 2,
    "members": true,
    "name": "Cannonball",
    "sp": 5,
    "overall_average": 195,
    "sell_average": 195,
    "buy_average": 195
  },
  "6": {
    "id": 6,
    "members": true,
    "name": "Cannon base",
    "sp": 187500,
    "overall_average": 179789,
    "sell_average": 177522,
    "buy_average": 176043
  }
}

To extract each item you would create a JSONObject of the whole String.

Then iterate over each of it's keys, these are the item ids e.g. (2, 6).

For each of those keys, get the value (this is the item's information), and then print the value of the "name" key out to a file.

 

Okay so just to clarify in simple terms, I need to sort the data into a more readable form then I need to get item id (the key), finally I need to just read the data from each key?

 

Thank you for the help it's much appreciated, btw I loved your OSBOT tutorial it really helped me get started with OSBOT and Java in general

 

[Explv]

2 minutes ago, flewis said:

 

Okay so just to clarify in simple terms, I need to sort the data into a more readable form then I need to get item id (the key), finally I need to just read the data from each key?

 

Thank you for the help it's much appreciated, btw I loved your OSBOT tutorial it really helped me get started with OSBOT and Java in general

 

You don't need to sort the data into a more readable form. I just did that to show you what it looks like.

Take a look at some JSON tutorials online, if you want to understand how to do it. JSON is very very simple.

If you just want the item names, check my above post.

Edited July 30, 2017 by Explv

[flewis]

6 minutes ago, Explv said:

 

If this is a just a one off thing though, I wrote a quick python script to do it:

import json

item_data = {}

with open('summary.json', 'r') as summary_file:
    item_data = json.load(summary_file)

item_names = [item_data[item_id]['name'] for item_id in item_data] 
    
with open('item_names.txt', 'w') as item_names_file:
    for item_name in item_names:
        item_names_file.write(item_name + "\n")

 

And here is the output:

https://hastebin.com/dewunamare.sql

 

I tried doing it in python but it's not a one off thing because the item prices update every couple of minutes and I wanted to use it to find good prices for items in the GE. 

[Explv]

2 minutes ago, flewis said:

 

I tried doing it in python but it's not a one off thing because the item prices update every couple of minutes and I wanted to use it to find good prices for items in the GE. 

If you want prices and stuff like that, I have already written a complete solution in the snippets section:

https://osbot.org/forum/topic/102611-ge-data-get-price-etc-by-item-name-no-external-libraries-required/

You can modify it to suit your needs accordingly.

 

Edited July 30, 2017 by Explv

[flewis]

6 minutes ago, Explv said:

If you want prices and stuff like that, I have already written a complete solution in the snippets section:

https://osbot.org/forum/topic/102611-ge-data-get-price-etc-by-item-name-no-external-libraries-required/

You can modify it to suit your needs accordingly.

 

 

Does it support buy and sell price ratio for profit?

[Explv]

1 minute ago, flewis said:

 

Does it support buy and sell price ratio for profit?

 

Right now it just stores the overall price value.

I can modify it though to store information. Will update the thread shortly

[flewis]

7 minutes ago, Explv said:

 

Right now it just stores the overall price value.

I can modify it though to store information. Will update the thread shortly

 

Thank you so much, saves me a lot of time

A quick suggestion is you should add all data to program because it's all really useful for making grand exchange scripts!!

[Tom]

If you have any plans to put a script on the SDN though, I'm pretty confident you cant use a JSON library unless you have the source in your script

[Explv]

25 minutes ago, Tom said:

If you have any plans to put a script on the SDN though, I'm pretty confident you cant use a JSON library unless you have the source in your script

Yeah, you can use the JSON Simple library no problem on the SDN. You just have to include the source (which is only a handful of classes).

 

55 minutes ago, flewis said:

 

Thank you so much, saves me a lot of time

A quick suggestion is you should add all data to program because it's all really useful for making grand exchange scripts!!

 

I have updated the original thread:

https://osbot.org/forum/topic/102611-ge-data-get-price-etc-by-item-name-no-external-libraries-required/

 

And also written a new one that uses the JSON Simple library (probably the preferred way)

https://osbot.org/forum/topic/126881-ge-data-using-rsbuddy-json-simple-library/