Grabbing a ton of prices for items

[Butters]

Was wondering if there's a good way to load a ton of item prices from websites in a speedy manner. Actually dunno if it's even possible without just checking the website whenever absolutely needed.

Hosting my own price database and refreshing it every now and then really isn't a decent option unless there's no other way. (say DB crashes)

Naturally, caching isn't an option here, cause prices change.

Say I want to load a 100 item prices on script start.

Atm using something like

final URL url = new URL("http://2007.runescape.wikia.com/wiki/Exchange:" + item.replaceAll(" ", "_"));
BufferedReader file = new BufferedReader(new InputStreamReader(url.openStream()));
// Parsing stuff

It takes more than a minute to load a ton of items, and probably would take longer on slow connections.

I remember when I used Perfect Fighter (script on SDN) it looked like it did price grabbing for loot rather quickly, though didn't test with a ton of items.

So, being naive here, but any reliable ways to do this and fast?

[dreameo]

What you need to do is pull all the data once and have it stored locally where then you can interact and pull stuff immediately. 

https://rsbuddy.com/exchange/summary.json

Be warned though, the JSON is dynamic and you can't just have a method that assumes the position of the elements (Assuming you are parsing this as a String and not JSON). 

[ThatGamerBlue]

Make a caching class, if a request comes in and the item isn't cached, get the price, store it in the cache, store the time with the price, when a request comes in check if the cached price is older than a day, if it is, recache, if it isn't return the cached price.

[Butters]

26 minutes ago, dreameo said:

What you need to do is pull all the data once and have it stored locally where then you can interact and pull stuff immediately. 

https://rsbuddy.com/exchange/summary.json

Be warned though, the JSON is dynamic and you can't just have a method that assumes the position of the elements (Assuming you are parsing this as a String and not JSON). 

Thanks! Didn't know about this. 

Couple of questions though:

1) What do you mean by "JSON is dynamic"? Is the structure bound to change?

2) Any provider which searches by item names and not ids?

3) Are RSbuddy prices accurate? Saw somewhere said that they're not.

3 minutes ago, ThatGamerBlue said:

Make a caching class, if a request comes in and the item isn't cached, get the price, store it in the cache, store the time with the price, when a request comes in check if the cached price is older than a day, if it is, recache, if it isn't return the cached price.

Nah always want to have the latest price, though if not, still a lot of checking.

Edited January 19, 2018 by nosepicker

[ThatGamerBlue]

3 minutes ago, nosepicker said:

Thanks! Didn't know about this. 

Couple of questions though:

1) What do you mean by "JSON is dynamic"? Is the structure bound to change?

2) Are RSbuddy prices accurate? Saw somewhere said that they're not.

Nah always want to have the latest price, though if not, still a lot of checking.

I don't know how long the GE takes between updates, but it's around daily, more often than that if the item is traded a lot like lobbies or highly farmed items, but imo this is the most efficient way to do it while keeping relatively up to date.

[dreameo]

8 minutes ago, nosepicker said:

Thanks! Didn't know about this. 

Couple of questions though:

1) What do you mean by "JSON is dynamic"? Is the structure bound to change?

2) Any provider which searches by item names and not ids?

3) Are RSbuddy prices accurate? Saw somewhere said that they're not.

Nah always want to have the latest price, though if not, still a lot of checking.

Yea, the properties in this JSON change index

You can search by item name, just look inside the JSON data

The prices are probably accurate 

[Butters]

5 minutes ago, dreameo said:

Yea, the properties in this JSON change index

You can search by item name, just look inside the JSON data

The prices are probably accurate 

Ah so RSBuddy likes to change property indexes, but not property names? If that's true then great.

Thanks

[Explv]

2 minutes ago, nosepicker said:

Ah so RSBuddy likes to change property indexes, but not property names? If that's true then great.

Thanks

Just use a JSON library like Json simple.

[Butters]

5 minutes ago, Explv said:

Just use a JSON library like Json simple.

Thanks, already am

[dreameo]

42 minutes ago, nosepicker said:

Ah so RSBuddy likes to change property indexes, but not property names? If that's true then great.

Thanks

Pastebin:

https://pastebin.com/RAnGfJ9Y

Usage:

  String itemName = "Iron ore";
  String property = "overall_average";

  String data = ItemData.getData(itemName, property);

 

You might have issues with external library, anyways, I just wrote this up. Note that, if you're looking for numerical values, you'd have to convert String to int.

 

Forget to mention that if you're making frequent calls to the same item, i'd be best if you made a map between item and the property. 

Edited January 19, 2018 by dreameo

[battleguard]

    public static OsBuddyItemConfig[] getItems() throws IOException {
        JsonParser parser = new JsonParser();
        JsonElement parse = parser.parse(readUrl("https://rsbuddy.com/exchange/summary.json"));
        JsonObject asJsonObject = parse.getAsJsonObject();
        Gson gson = new Gson();
        Set<Map.Entry<String, JsonElement>> entries = asJsonObject.entrySet();
        Iterator<Map.Entry<String, JsonElement>> iterator = entries.iterator();
        OsBuddyItemConfig[] items = new OsBuddyItemConfig[entries.size()];
        for(int i = 0; i < entries.size(); i++)
        {
            JsonElement value = iterator.next().getValue();
            OsBuddyItemConfig osBuddyItemConfig = gson.fromJson(value, OsBuddyItemConfig.class);
            items[i] = osBuddyItemConfig;
        }
        return items;
    }

    private static String readUrl(final String urlPath) throws IOException {
        URL url = new URL(urlPath);
        URLConnection con = url.openConnection();
        con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36");
        con.setUseCaches(true);
        BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String json = br.readLine();
        br.close();
        return json;
    }

    public class OsBuddyItemConfig
    {
        public int id;
        public String name;
        public int buy_average;
        public int sp;
        public int overall_average;
        public boolean members;
        public int sell_average;
    }

Here is what I have been using lately, but this requires you to extract GSON's source into your script, but its so nice and clean and I use many json classes so rolling my own parser just was not worth it.

Edited January 19, 2018 by battleguard

[Butters]

Thank you guys for the replies.

I ended up using JSON minimal https://eclipsesource.com/blogs/2013/04/18/minimal-json-parser-for-java/

At least the writers claim that it has better performance than others.

Example code if someones interested

private final static String RSBUDDY_URL = "https://rsbuddy.com/exchange/summary.json";

public static HashMap<String, Integer> getPriceMap(List<String> items) {
		HashMap<String, Integer> priceMap = new HashMap<>();

		try {
			URL url = new URL(RSBUDDY_URL);
			BufferedReader jsonFile = new BufferedReader(new InputStreamReader(url.openStream()));
			JsonObject priceJSON = JsonObject.readFrom(jsonFile.readLine());
			Iterator<Member> iterator = priceJSON.iterator();
			
			while (iterator.hasNext()) {
				JsonObject itemJSON = priceJSON.get(iterator.next().getName()).asObject();
				String itemName = itemJSON.get("name").asString();
				if (items.contains(itemName)) {
					priceMap.put(itemName, itemJSON.get("buy_average").asInt());
				}
			}
		} catch (Exception e) {
			log("Failed to grab item prices!");
		}
		return priceMap;
	}

Usage

List<String> itemList = Arrays.asList("Rune platebody", "Bronze axe", "Gold bar", "Air rune", "Fire rune", "Amulet of strength");
getPriceMap(itemList);

 

P.S. total count of RS buddy item data is 3412.

Are there really only that many items in osrs?

Edited January 19, 2018 by nosepicker

[liverare]

OSBuddy are able to record Grand Exchange transactions straight from their client, so their prices are factual. However, not everybody uses OSBuddy. If you're making a GE flipping script, then I'd suggest you pool price information from as many sources as possible, especially from Jagex themselves. It can't get anymore accurate than the actual RS GE website.

[The Undefeated]

2 hours ago, liverare said:

OSBuddy are able to record Grand Exchange transactions straight from their client, so their prices are factual. However, not everybody uses OSBuddy. If you're making a GE flipping script, then I'd suggest you pool price information from as many sources as possible, especially from Jagex themselves. It can't get anymore accurate than the actual RS GE website.

That's not true. The RS GE website doesn't update that frequently and their prices are always more off than those from OSBuddy.

I really don't recommend scraping, especially when loading more than 5 items since it will take some time to finish the scraping process.