flewis Posted July 30, 2017 Share Posted July 30, 2017 Dear all, I have found some data on a website that I'd like to extract. In java I don't know how to get the data from the website and sort it into a file in my scripts folder. For example the website has data like: "2": {"buy_average": 193, "members": true, "name": "Cannonball", "sp": 5, "overall_average": 195, "sell_average": 194, "id": 2}, "6": {"buy_average": 174492, "members": true, "name": "Cannon base", "sp": 187500, "overall_average": 180888, "sell_average": 176566, "id": 6}, Only problem is it's all in one string and I need to separate each item out onto a different line in the text file (the file contains all Runescape items). Website I want to extract data from: https://rsbuddy.com/exchange/summary.json Sorry I'm pretty new to Java so I have no idea what I'm doing tbh xD Thank you for any help though!! -Flewis Quote Link to comment Share on other sites More sharing options...
Explv Posted July 30, 2017 Share Posted July 30, 2017 Firstly that file will not include all RuneScape items, only ones tradeable on the Grand Exchange. It is a JSON file, you should use a JSON library to handle the data, such as https://github.com/fangyidong/json-simple or https://github.com/google/gson Here is what the file looks like prettified: { "2": { "id": 2, "members": true, "name": "Cannonball", "sp": 5, "overall_average": 195, "sell_average": 195, "buy_average": 195 }, "6": { "id": 6, "members": true, "name": "Cannon base", "sp": 187500, "overall_average": 179789, "sell_average": 177522, "buy_average": 176043 } } To extract each item you would create a JSONObject of the whole String. Then iterate over each of it's keys, these are the item ids e.g. (2, 6). For each of those keys, get the value (this is the item's information), and then print the value of the "name" key out to a file. 1 Quote Link to comment Share on other sites More sharing options...
Explv Posted July 30, 2017 Share Posted July 30, 2017 (edited) 12 minutes ago, flewis said: Dear all, I have found some data on a website that I'd like to extract. In java I don't know how to get the data from the website and sort it into a file in my scripts folder. For example the website has data like: "2": {"buy_average": 193, "members": true, "name": "Cannonball", "sp": 5, "overall_average": 195, "sell_average": 194, "id": 2}, "6": {"buy_average": 174492, "members": true, "name": "Cannon base", "sp": 187500, "overall_average": 180888, "sell_average": 176566, "id": 6}, Only problem is it's all in one string and I need to separate each item out onto a different line in the text file (the file contains all Runescape items). Website I want to extract data from: https://rsbuddy.com/exchange/summary.json Sorry I'm pretty new to Java so I have no idea what I'm doing tbh xD Thank you for any help though!! -Flewis If this is a just a one off thing though, I wrote a quick python script to do it: import json item_data = {} with open('summary.json', 'r') as summary_file: item_data = json.load(summary_file) item_names = [item_data[item_id]['name'] for item_id in item_data] with open('item_names.txt', 'w') as item_names_file: for item_name in item_names: item_names_file.write(item_name + "\n") And here is the output: https://hastebin.com/dewunamare.sql Edited July 30, 2017 by Explv 2 Quote Link to comment Share on other sites More sharing options...
flewis Posted July 30, 2017 Author Share Posted July 30, 2017 2 minutes ago, Explv said: Firstly that file will not include all RuneScape items, only ones tradeable on the Grand Exchange. It is a JSON file, you should use a JSON library to handle the data, such as https://github.com/fangyidong/json-simple or https://github.com/google/gson Here is what the file looks like prettified: { "2": { "id": 2, "members": true, "name": "Cannonball", "sp": 5, "overall_average": 195, "sell_average": 195, "buy_average": 195 }, "6": { "id": 6, "members": true, "name": "Cannon base", "sp": 187500, "overall_average": 179789, "sell_average": 177522, "buy_average": 176043 } } To extract each item you would create a JSONObject of the whole String. Then iterate over each of it's keys, these are the item ids e.g. (2, 6). For each of those keys, get the value (this is the item's information), and then print the value of the "name" key out to a file. Okay so just to clarify in simple terms, I need to sort the data into a more readable form then I need to get item id (the key), finally I need to just read the data from each key? Thank you for the help it's much appreciated, btw I loved your OSBOT tutorial it really helped me get started with OSBOT and Java in general Quote Link to comment Share on other sites More sharing options...
Explv Posted July 30, 2017 Share Posted July 30, 2017 (edited) 2 minutes ago, flewis said: Okay so just to clarify in simple terms, I need to sort the data into a more readable form then I need to get item id (the key), finally I need to just read the data from each key? Thank you for the help it's much appreciated, btw I loved your OSBOT tutorial it really helped me get started with OSBOT and Java in general You don't need to sort the data into a more readable form. I just did that to show you what it looks like. Take a look at some JSON tutorials online, if you want to understand how to do it. JSON is very very simple. If you just want the item names, check my above post. Edited July 30, 2017 by Explv Quote Link to comment Share on other sites More sharing options...
flewis Posted July 30, 2017 Author Share Posted July 30, 2017 6 minutes ago, Explv said: If this is a just a one off thing though, I wrote a quick python script to do it: import json item_data = {} with open('summary.json', 'r') as summary_file: item_data = json.load(summary_file) item_names = [item_data[item_id]['name'] for item_id in item_data] with open('item_names.txt', 'w') as item_names_file: for item_name in item_names: item_names_file.write(item_name + "\n") And here is the output: https://hastebin.com/dewunamare.sql I tried doing it in python but it's not a one off thing because the item prices update every couple of minutes and I wanted to use it to find good prices for items in the GE. Quote Link to comment Share on other sites More sharing options...
Explv Posted July 30, 2017 Share Posted July 30, 2017 (edited) 2 minutes ago, flewis said: I tried doing it in python but it's not a one off thing because the item prices update every couple of minutes and I wanted to use it to find good prices for items in the GE. If you want prices and stuff like that, I have already written a complete solution in the snippets section: https://osbot.org/forum/topic/102611-ge-data-get-price-etc-by-item-name-no-external-libraries-required/ You can modify it to suit your needs accordingly. Edited July 30, 2017 by Explv 1 Quote Link to comment Share on other sites More sharing options...
flewis Posted July 30, 2017 Author Share Posted July 30, 2017 6 minutes ago, Explv said: If you want prices and stuff like that, I have already written a complete solution in the snippets section: https://osbot.org/forum/topic/102611-ge-data-get-price-etc-by-item-name-no-external-libraries-required/ You can modify it to suit your needs accordingly. Does it support buy and sell price ratio for profit? Quote Link to comment Share on other sites More sharing options...
Explv Posted July 30, 2017 Share Posted July 30, 2017 1 minute ago, flewis said: Does it support buy and sell price ratio for profit? Right now it just stores the overall price value. I can modify it though to store information. Will update the thread shortly 1 Quote Link to comment Share on other sites More sharing options...
flewis Posted July 30, 2017 Author Share Posted July 30, 2017 7 minutes ago, Explv said: Right now it just stores the overall price value. I can modify it though to store information. Will update the thread shortly Thank you so much, saves me a lot of time A quick suggestion is you should add all data to program because it's all really useful for making grand exchange scripts!! Quote Link to comment Share on other sites More sharing options...
Tom Posted July 30, 2017 Share Posted July 30, 2017 If you have any plans to put a script on the SDN though, I'm pretty confident you cant use a JSON library unless you have the source in your script Quote Link to comment Share on other sites More sharing options...
Explv Posted July 30, 2017 Share Posted July 30, 2017 25 minutes ago, Tom said: If you have any plans to put a script on the SDN though, I'm pretty confident you cant use a JSON library unless you have the source in your script Yeah, you can use the JSON Simple library no problem on the SDN. You just have to include the source (which is only a handful of classes). 55 minutes ago, flewis said: Thank you so much, saves me a lot of time A quick suggestion is you should add all data to program because it's all really useful for making grand exchange scripts!! I have updated the original thread: https://osbot.org/forum/topic/102611-ge-data-get-price-etc-by-item-name-no-external-libraries-required/ And also written a new one that uses the JSON Simple library (probably the preferred way) https://osbot.org/forum/topic/126881-ge-data-using-rsbuddy-json-simple-library/ Quote Link to comment Share on other sites More sharing options...