Jump to content

Grabbing a ton of prices for items


Butters

Recommended Posts

Was wondering if there's a good way to load a ton of item prices from websites in a speedy manner. Actually dunno if it's even possible without just checking the website whenever absolutely needed.

Hosting my own price database and refreshing it every now and then really isn't a decent option unless there's no other way. (say DB crashes)

Naturally, caching isn't an option here, cause prices change.

Say I want to load a 100 item prices on script start.

Atm using something like

final URL url = new URL("http://2007.runescape.wikia.com/wiki/Exchange:" + item.replaceAll(" ", "_"));
BufferedReader file = new BufferedReader(new InputStreamReader(url.openStream()));
// Parsing stuff

It takes more than a minute to load a ton of items, and probably would take longer on slow connections.

I remember when I used Perfect Fighter (script on SDN) it looked like it did price grabbing for loot rather quickly, though didn't test with a ton of items.

So, being naive here, but any reliable ways to do this and fast?

Link to comment
Share on other sites

What you need to do is pull all the data once and have it stored locally where then you can interact and pull stuff immediately. 

https://rsbuddy.com/exchange/summary.json

Be warned though, the JSON is dynamic and you can't just have a method that assumes the position of the elements (Assuming you are parsing this as a String and not JSON). 

  • Like 1
Link to comment
Share on other sites

26 minutes ago, dreameo said:

What you need to do is pull all the data once and have it stored locally where then you can interact and pull stuff immediately. 

https://rsbuddy.com/exchange/summary.json

Be warned though, the JSON is dynamic and you can't just have a method that assumes the position of the elements (Assuming you are parsing this as a String and not JSON). 

Thanks! Didn't know about this. 

Couple of questions though:

1) What do you mean by "JSON is dynamic"? Is the structure bound to change?

2) Any provider which searches by item names and not ids?

3) Are RSbuddy prices accurate? Saw somewhere said that they're not.

3 minutes ago, ThatGamerBlue said:

Make a caching class, if a request comes in and the item isn't cached, get the price, store it in the cache, store the time with the price, when a request comes in check if the cached price is older than a day, if it is, recache, if it isn't return the cached price.

Nah always want to have the latest price, though if not, still a lot of checking.

Edited by nosepicker
Link to comment
Share on other sites

3 minutes ago, nosepicker said:

Thanks! Didn't know about this. 

Couple of questions though:

1) What do you mean by "JSON is dynamic"? Is the structure bound to change?

2) Are RSbuddy prices accurate? Saw somewhere said that they're not.

Nah always want to have the latest price, though if not, still a lot of checking.

I don't know how long the GE takes between updates, but it's around daily, more often than that if the item is traded a lot like lobbies or highly farmed items, but imo this is the most efficient way to do it while keeping relatively up to date.

Link to comment
Share on other sites

8 minutes ago, nosepicker said:

Thanks! Didn't know about this. 

Couple of questions though:

1) What do you mean by "JSON is dynamic"? Is the structure bound to change?

2) Any provider which searches by item names and not ids?

3) Are RSbuddy prices accurate? Saw somewhere said that they're not.

Nah always want to have the latest price, though if not, still a lot of checking.

Yea, the properties in this JSON change index

You can search by item name, just look inside the JSON data

The prices are probably accurate 

  • Like 1
Link to comment
Share on other sites

42 minutes ago, nosepicker said:

Ah so RSBuddy likes to change property indexes, but not property names? If that's true then great.

Thanks

Pastebin:

https://pastebin.com/RAnGfJ9Y

Usage:

  String itemName = "Iron ore";
  String property = "overall_average";

  String data = ItemData.getData(itemName, property);

 

You might have issues with external library, anyways, I just wrote this up. Note that, if you're looking for numerical values, you'd have to convert String to int.

 

Forget to mention that if you're making frequent calls to the same item, i'd be best if you made a map between item and the property. 

Edited by dreameo
  • Like 1
Link to comment
Share on other sites

    public static OsBuddyItemConfig[] getItems() throws IOException {
        JsonParser parser = new JsonParser();
        JsonElement parse = parser.parse(readUrl("https://rsbuddy.com/exchange/summary.json"));
        JsonObject asJsonObject = parse.getAsJsonObject();
        Gson gson = new Gson();
        Set<Map.Entry<String, JsonElement>> entries = asJsonObject.entrySet();
        Iterator<Map.Entry<String, JsonElement>> iterator = entries.iterator();
        OsBuddyItemConfig[] items = new OsBuddyItemConfig[entries.size()];
        for(int i = 0; i < entries.size(); i++)
        {
            JsonElement value = iterator.next().getValue();
            OsBuddyItemConfig osBuddyItemConfig = gson.fromJson(value, OsBuddyItemConfig.class);
            items[i] = osBuddyItemConfig;
        }
        return items;
    }

    private static String readUrl(final String urlPath) throws IOException {
        URL url = new URL(urlPath);
        URLConnection con = url.openConnection();
        con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36");
        con.setUseCaches(true);
        BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream()));
        String json = br.readLine();
        br.close();
        return json;
    }

    public class OsBuddyItemConfig
    {
        public int id;
        public String name;
        public int buy_average;
        public int sp;
        public int overall_average;
        public boolean members;
        public int sell_average;
    }

Here is what I have been using lately, but this requires you to extract GSON's source into your script, but its so nice and clean and I use many json classes so rolling my own parser just was not worth it.

Edited by battleguard
  • Like 1
Link to comment
Share on other sites

Thank you guys for the replies.

I ended up using JSON minimal https://eclipsesource.com/blogs/2013/04/18/minimal-json-parser-for-java/

At least the writers claim that it has better performance than others.

Example code if someones interested

private final static String RSBUDDY_URL = "https://rsbuddy.com/exchange/summary.json";

public static HashMap<String, Integer> getPriceMap(List<String> items) {
		HashMap<String, Integer> priceMap = new HashMap<>();

		try {
			URL url = new URL(RSBUDDY_URL);
			BufferedReader jsonFile = new BufferedReader(new InputStreamReader(url.openStream()));
			JsonObject priceJSON = JsonObject.readFrom(jsonFile.readLine());
			Iterator<Member> iterator = priceJSON.iterator();
			
			while (iterator.hasNext()) {
				JsonObject itemJSON = priceJSON.get(iterator.next().getName()).asObject();
				String itemName = itemJSON.get("name").asString();
				if (items.contains(itemName)) {
					priceMap.put(itemName, itemJSON.get("buy_average").asInt());
				}
			}
		} catch (Exception e) {
			log("Failed to grab item prices!");
		}
		return priceMap;
	}

Usage

List<String> itemList = Arrays.asList("Rune platebody", "Bronze axe", "Gold bar", "Air rune", "Fire rune", "Amulet of strength");
getPriceMap(itemList);

 

P.S. total count of RS buddy item data is 3412.

Are there really only that many items in osrs? :D

Edited by nosepicker
Link to comment
Share on other sites

OSBuddy are able to record Grand Exchange transactions straight from their client, so their prices are factual. However, not everybody uses OSBuddy. If you're making a GE flipping script, then I'd suggest you pool price information from as many sources as possible, especially from Jagex themselves. It can't get anymore accurate than the actual RS GE website.

Link to comment
Share on other sites

2 hours ago, liverare said:

OSBuddy are able to record Grand Exchange transactions straight from their client, so their prices are factual. However, not everybody uses OSBuddy. If you're making a GE flipping script, then I'd suggest you pool price information from as many sources as possible, especially from Jagex themselves. It can't get anymore accurate than the actual RS GE website.

That's not true. The RS GE website doesn't update that frequently and their prices are always more off than those from OSBuddy.

I really don't recommend scraping, especially when loading more than 5 items since it will take some time to finish the scraping process.

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...