RSBuddy Exchange Oracle

[liverare]

This oracle processes the the summary.json from the RSBuddy Exchange website.

public static void main(String[] args) throws IOException {
	
	// Load prices for now
	RSBuddyExchangeOracle.retrievePriceGuide();
  	// Load prices from yesterday
	RSBuddyExchangeOracle.retrievePriceGuide(System.currentTimeMillis() - (24 * 60 * 60 * 1000));

	// ...
}

All the prices will be processed and cached, then accessed like:

RSBuddyExchangePrice price;
List<RSBuddyExchangePrice> prices;

// Get item by name
price = RSBuddyExchangeOracle.getItemByName("Dragon chainbody");
System.out.println(price);

// Get item by ID
price = RSBuddyExchangeOracle.getItemByID(4151); // Abyssal whip
System.out.println(price);

// Get item(s) by name containing
price = RSBuddyExchangeOracle.getItemByNameContaining("Bronze dagg"); // Bronze dagger
System.out.println(price);
		
prices = RSBuddyExchangeOracle.getItemsByNameContaining("Bronze dagg"); // [ Bronze dagger, Bronze dagger(p++), Bronze dagger(p+), Bronze dagger(p) ]
System.out.println(prices);

// Get item(s) by name matching
price = RSBuddyExchangeOracle.getItemByNameMatching("Rune kiteshield \\(.?\\)"); // Rune kiteshield (g)
System.out.println(price);

prices = RSBuddyExchangeOracle.getItemsByNameMatching("Rune kiteshield \\(.?\\)"); // [ Rune kiteshield (g), Rune kiteshield (t) ]
System.out.println(prices);

Running these returns the following:

Quote

{ "id":3140, "name":"Dragon chainbody", "members":true, "storePrice":250000, "buyPrice":1326637, "buyQuantity":19, "sellPrice":1316888, "sellQuantity":50, "overallPrice":1319572, "overallQuantity":69 }
{ "id":4151, "name":"Abyssal whip", "members":true, "storePrice":120001, "buyPrice":2895692, "buyQuantity":52, "sellPrice":2892661, "sellQuantity":72, "overallPrice":2893932, "overallQuantity":124 }
{ "id":1205, "name":"Bronze dagger", "members":false, "storePrice":10, "buyPrice":6, "buyQuantity":11, "sellPrice":6, "sellQuantity":38, "overallPrice":6, "overallQuantity":49 }
[{ "id":1205, "name":"Bronze dagger", "members":false, "storePrice":10, "buyPrice":6, "buyQuantity":11, "sellPrice":6, "sellQuantity":38, "overallPrice":6, "overallQuantity":49 }, { "id":5688, "name":"Bronze dagger(p++)", "members":true, "storePrice":10, "buyPrice":0, "buyQuantity":0, "sellPrice":0, "sellQuantity":0, "overallPrice":0, "overallQuantity":0 }, { "id":5670, "name":"Bronze dagger(p+)", "members":true, "storePrice":10, "buyPrice":0, "buyQuantity":0, "sellPrice":0, "sellQuantity":0, "overallPrice":0, "overallQuantity":0 }, { "id":1221, "name":"Bronze dagger(p)", "members":true, "storePrice":10, "buyPrice":0, "buyQuantity":0, "sellPrice":0, "sellQuantity":0, "overallPrice":0, "overallQuantity":0 }]
{ "id":2621, "name":"Rune kiteshield (g)", "members":false, "storePrice":54400, "buyPrice":254498, "buyQuantity":2, "sellPrice":243753, "sellQuantity":3, "overallPrice":248051, "overallQuantity":5 }
[{ "id":2621, "name":"Rune kiteshield (g)", "members":false, "storePrice":54400, "buyPrice":254498, "buyQuantity":2, "sellPrice":243753, "sellQuantity":3, "overallPrice":248051, "overallQuantity":5 }, { "id":2629, "name":"Rune kiteshield (t)", "members":false, "storePrice":54400, "buyPrice":72942, "buyQuantity":3, "sellPrice":70538, "sellQuantity":4, "overallPrice":71568, "overallQuantity":7 }]

Have fun!

RSBuddyExchangeOracle.java

RSBuddyExchangePrice.java

Test.java

Edited December 30, 2018 by liverare

[Naked]

You're a dime.

[flvshcvrds]

neat share, thanks!

[Cod3xDev]

Nice! I'll definitely integrate this into my scripts instead of using a static price for logs/ore  

[liverare]

I've fixed the RSBuddyExchangePrice.java, as it was missing a toString method.

[NoxMerc]

/

Edited February 6, 2019 by NoxMerc

[liverare]

9 hours ago, NoxMerc said:

I'm not sure why, but using WeakHashMap for the JSON_BY_IDS field is causing the map to only go up to 254 or so fields.

 

This is causing my method calls to getItemById to fail.

Testing 434
null
Testing 532
null

Testing 434
{ "id":434, "name":"Clay", "members":false, "storePrice":1, "buyPrice":53, "buyQuantity":13236, "sellPrice":52, "sellQuantity":31, "overallPrice":53, "overallQuantity":13267 }
Testing 532
{ "id":532, "name":"Big bones", "members":false, "storePrice":1, "buyPrice":274, "buyQuantity":3140, "sellPrice":268, "sellQuantity":4256, "overallPrice":270, "overallQuantity":7396 }

 

Try replacing it with a HashMap.

[NoxMerc]

I...I did. That's what the second picture proves.

[liverare]

1 hour ago, NoxMerc said:

I...I did. That's what the second picture proves.

Sorry, it's New Years and I'm pretty tired lol.

I'm not sure why you only got 254 mapped entries when I got the following:

        System.out.println("JSON_BY_IDS.size() = " + JSON_BY_IDS.size());
        System.out.println("JSON_BY_NAMES.size() = " + JSON_BY_NAMES.size());

Which resulted with:

Quote

JSON_BY_IDS.size() = 3478
JSON_BY_NAMES.size() = 3414

I was able to get prices for items with ids 434 and 512.

However, the discrepancy between the number of result in the two maps is because JSON_BY_NAMES stores Strings as keys and there are multiple items sharing the exact same names:

Quote

Priest gown
Bandana eyepatch
Bandana eyepatch
Bandana eyepatch
Mith grapple
Asgarnian ale
Greenman's ale
Dragon bitter
Yak-hide armour
Karambwan vessel
Myre snelm
Blood'n'tar snelm
Ochre snelm
Bruise blue snelm
Blamish myre shell
Blamish red shell
Blamish ochre shell
Blamish blue shell
Candle lantern
Woven top
Woven top
Shirt
Shirt
Trousers
Trousers
Shorts
Shorts
Skirt
Skirt
Chef's delight
Broodoo shield (10)
Broodoo shield
Broodoo shield (10)
Broodoo shield
Tribal mask
Tribal mask
Tribal top
Villager robe
Villager hat
Villager sandals
Villager armband
Tribal top
Villager robe
Villager hat
Villager sandals
Villager armband
Tribal top
Villager robe
Villager hat
Villager sandals
Villager armband
Desert top
Sweetcorn
Stripy pirate shirt
Pirate bandana
Pirate leggings
Stripy pirate shirt
Pirate bandana
Pirate leggings
Stripy pirate shirt
Pirate bandana
Pirate leggings
Cooked chompy
Snake hide

Thanks for highlighting this problem, you've definitely found a bug. However, fixing this would either require a data restructure or query restructure, which is either more complicated or less efficient. Also, as long as you can continue to get items by IDs, then I'm not too worried.

Edited January 1, 2019 by liverare

[NoxMerc]

1 hour ago, liverare said:

Sorry, it's New Years and I'm pretty tired lol.

All good!

Here is what I'm using to test.

public class RSBuddyExchangeOracleTester {
    private static final String[] NAMES_TO_TEST = new String[] { "Clay", "Adamant pickaxe", "Rune axe", "Mithril platebody" };
    private static final int[] IDS_TO_TEST = new int[] { 434,  532 }; // clay, big bones
    public static void main(String[] args) throws IOException {
        System.out.println("Testing RSBuddy Exchange Oracle");
        long start = System.currentTimeMillis();
        RSBuddyExchangeOracle.retrievePriceGuide();
        System.out.println("Took " + (System.currentTimeMillis() - start) + "ms to load prices");
        for(int i : IDS_TO_TEST) {
            System.out.println("Testing " + i);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByID(i);
            System.out.println(price);
        }
        for(String s: NAMES_TO_TEST) {
            System.out.println("Testing " + s);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByName(s);
            System.out.println(price);
        }

        System.out.println("Total time taken: " + (System.currentTimeMillis() - start) + "ms");
    }
}

With WeakHashMap, the two ID tests consistently return null. With HashMap, they work out okay. Am I misusing the Oracle in some way?

Edited January 1, 2019 by NoxMerc

[liverare]

22 hours ago, NoxMerc said:

All good!

Here is what I'm using to test.

public class RSBuddyExchangeOracleTester {
    private static final String[] NAMES_TO_TEST = new String[] { "Clay", "Adamant pickaxe", "Rune axe", "Mithril platebody" };
    private static final int[] IDS_TO_TEST = new int[] { 434,  532 }; // clay, big bones
    public static void main(String[] args) throws IOException {
        System.out.println("Testing RSBuddy Exchange Oracle");
        long start = System.currentTimeMillis();
        RSBuddyExchangeOracle.retrievePriceGuide();
        System.out.println("Took " + (System.currentTimeMillis() - start) + "ms to load prices");
        for(int i : IDS_TO_TEST) {
            System.out.println("Testing " + i);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByID(i);
            System.out.println(price);
        }
        for(String s: NAMES_TO_TEST) {
            System.out.println("Testing " + s);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByName(s);
            System.out.println(price);
        }

        System.out.println("Total time taken: " + (System.currentTimeMillis() - start) + "ms");
    }
}

 With WeakHashMap, the two ID tests consistently return null. With HashMap, they work out okay. Am I misusing the Oracle in some way?

Strange. I ran your tests without having changed the oracle in any way, and my results are:

Quote

Testing RSBuddy Exchange Oracle
Took 757ms to load prices
Testing 434
{ "id":434, "name":"Clay", "members":false, "storePrice":1, "buyPrice":65, "buyQuantity":16804, "sellPrice":64, "sellQuantity":486, "overallPrice":65, "overallQuantity":17290 }
Testing 532
{ "id":532, "name":"Big bones", "members":false, "storePrice":1, "buyPrice":267, "buyQuantity":7816, "sellPrice":265, "sellQuantity":9912, "overallPrice":266, "overallQuantity":17728 }
Testing Clay
{ "id":434, "name":"Clay", "members":false, "storePrice":1, "buyPrice":65, "buyQuantity":16804, "sellPrice":64, "sellQuantity":486, "overallPrice":65, "overallQuantity":17290 }
Testing Adamant pickaxe
{ "id":1271, "name":"Adamant pickaxe", "members":false, "storePrice":3200, "buyPrice":3759, "buyQuantity":36, "sellPrice":3547, "sellQuantity":20, "overallPrice":3683, "overallQuantity":56 }
Testing Rune axe
{ "id":1359, "name":"Rune axe", "members":false, "storePrice":12800, "buyPrice":7319, "buyQuantity":155, "sellPrice":7292, "sellQuantity":157, "overallPrice":7305, "overallQuantity":312 }
Testing Mithril platebody
{ "id":1121, "name":"Mithril platebody", "members":false, "storePrice":5200, "buyPrice":2839, "buyQuantity":643, "sellPrice":2767, "sellQuantity":1541, "overallPrice":2788, "overallQuantity":2184 }
Total time taken: 767ms

If you're getting null, it could perhaps be a caching issue, connection issue, or something else. Try again with Eclipse.

 

[NoxMerc]

Is it because I have less RAM (16Gb, but I'm always using most of it) available and the JVM collects the WeakReferences from the WeakHashMap sooner because it's trying to free up memory, whereas with your machine the JVM isn't collecting?

 

*Edit

Further testing confirms this. Tested on my laptop with 32Gb RAM.

public class RSBuddyExchangeOracleTester {
    private static final String[] NAMES_TO_TEST = new String[] { "Clay", "Adamant pickaxe", "Rune axe", "Mithril platebody" };
    private static final int[] IDS_TO_TEST = new int[] { 434,  532 }; // clay, big bones
    public static void main(String[] args) throws IOException {
        System.out.println("Testing RSBuddy Exchange Oracle");
        long start = System.currentTimeMillis();
        RSBuddyExchangeOracle.retrievePriceGuide();
        System.out.println("Took " + (System.currentTimeMillis() - start) + "ms to load prices");
        System.gc(); // <-------------- IMPORTANT
        for(int i : IDS_TO_TEST) {
            System.out.println("Testing " + i);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByID(i);
            System.out.println(price);
        }
        for(String s: NAMES_TO_TEST) {
            System.out.println("Testing " + s);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByName(s);
            System.out.println(price);
        }

        System.out.println("Total time taken: " + (System.currentTimeMillis() - start) + "ms");
    }
}

Without System.gc(), things run as normal. Running System.gc() will cause the WeakHashMap to collect its Integer values. I'm not immediately sure why the String values aren't collected.

Edited January 2, 2019 by NoxMerc

[liverare]

13 hours ago, NoxMerc said:

Is it because I have less RAM (16Gb, but I'm always using most of it) available and the JVM collects the WeakReferences from the WeakHashMap sooner because it's trying to free up memory, whereas with your machine the JVM isn't collecting?

 

*Edit

Further testing confirms this. Tested on my laptop with 32Gb RAM.

public class RSBuddyExchangeOracleTester {
    private static final String[] NAMES_TO_TEST = new String[] { "Clay", "Adamant pickaxe", "Rune axe", "Mithril platebody" };
    private static final int[] IDS_TO_TEST = new int[] { 434,  532 }; // clay, big bones
    public static void main(String[] args) throws IOException {
        System.out.println("Testing RSBuddy Exchange Oracle");
        long start = System.currentTimeMillis();
        RSBuddyExchangeOracle.retrievePriceGuide();
        System.out.println("Took " + (System.currentTimeMillis() - start) + "ms to load prices");
        System.gc(); // <-------------- IMPORTANT
        for(int i : IDS_TO_TEST) {
            System.out.println("Testing " + i);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByID(i);
            System.out.println(price);
        }
        for(String s: NAMES_TO_TEST) {
            System.out.println("Testing " + s);
            RSBuddyExchangePrice price = RSBuddyExchangeOracle.getItemByName(s);
            System.out.println(price);
        }

        System.out.println("Total time taken: " + (System.currentTimeMillis() - start) + "ms");
    }
}

Without System.gc(), things run as normal. Running System.gc() will cause the WeakHashMap to collect its Integer values. I'm not immediately sure why the String values aren't collected.

If I recall, Strings aren't cleared up by the garbage collector because the values are stored separately and as constants (because strings can be stupidly long and so memory intensive, so it's best to reuse wherever possible).

I used WeakHashMap because "it discards entries when the key is no longer strongly reachable from live code" which helps manage memory, which is important when you've got like 4K entries and you're not going to be using them all. However, I forgot strings don't get removed, so the Map<String, ...> might as well be a HashMap.

Edited January 3, 2019 by liverare

[NoxMerc]

I suppose I don't much see the purpose of even having a cache if it's randomly going to render itself unusable at some point. Especially when some point is directly after initializing the cache.

Anyway I did some experimenting. To reduce memory footprint I changed RSBuddyExchangePrice from containing an internal Map<String, Object> to having distinct fields. This caused no difference in performance, however it should reduce the amount of memory each RSBEP uses by at least (110 * 2 + 32*10),, or just about 2Mb for all 4,000 (where 110 is the length of the json string and there are 10 string references).

Spoiler

**Note: The difference in time has little to do with the structural changes of the RSBuddyExchangePrice and more to do with moving the compilation of the regex pattern out of parse() and into processIntoJSONMap()

Testing RSBuddy Exchange Oracle Mapper - ItemJson Map<String, Object>
Total time taken: 10327ms
Average time taken: 16ms
Min time taken: Optional[14]ms
Max time taken: Optional[102]ms

Testing RSBuddy Exchange Oracle Mapper - Distinct Fields
Total time taken: 9530ms
Average time taken: 8ms
Min time taken: Optional[7]ms
Max time taken: Optional[36]ms

The second thing I did is that remove the Pattern.compile() call from the parse(String jsonstr) method to its calling method. I think it only needs to be called once, but the implementation was compiling it for each new item. This cut the time in half it takes to parse all 4000 or so odd items (which, mind you, is going from on average 16ms to 8ms...so..not a huge deal).

Lastly, I changed the return values from longs to ints. This might be questionable, but I don't see any of the fields possibly breaking the int maxvalue.

This tool saved me from having to map IDs manually (I prefer to do everything by name), so thanks so much for posting it when you did. I'm not trying to be a jerk or anything with these suggestions, just in the pursuit of making awesome things more awesome when I can.

RSBuddyExchangeOracle.java

RSBuddyExchangePrice.java

Edited January 3, 2019 by NoxMerc

[liverare]

3 hours ago, NoxMerc said:

I suppose I don't much see the purpose of even having a cache if it's randomly going to render itself unusable at some point. Especially when some point is directly after initializing the cache.

Anyway I did some experimenting. To reduce memory footprint I changed RSBuddyExchangePrice from containing an internal Map<String, Object> to having distinct fields. This caused no difference in performance, however it should reduce the amount of memory each RSBEP uses by at least (110 * 2 + 32*10),, or just about 2Mb for all 4,000 (where 110 is the length of the json string and there are 10 string references).

   Reveal hidden contents

**Note: The difference in time has little to do with the structural changes of the RSBuddyExchangePrice and more to do with moving the compilation of the regex pattern out of parse() and into processIntoJSONMap()

Testing RSBuddy Exchange Oracle Mapper - ItemJson Map<String, Object>
Total time taken: 10327ms
Average time taken: 16ms
Min time taken: Optional[14]ms
Max time taken: Optional[102]ms

Testing RSBuddy Exchange Oracle Mapper - Distinct Fields
Total time taken: 9530ms
Average time taken: 8ms
Min time taken: Optional[7]ms
Max time taken: Optional[36]ms

The second thing I did is that remove the Pattern.compile() call from the parse(String jsonstr) method to its calling method. I think it only needs to be called once, but the implementation was compiling it for each new item. This cut the time in half it takes to parse all 4000 or so odd items (which, mind you, is going from on average 16ms to 8ms...so..not a huge deal).

Lastly, I changed the return values from longs to ints. This might be questionable, but I don't see any of the fields possibly breaking the int maxvalue.

 This tool saved me from having to map IDs manually (I prefer to do everything by name), so thanks so much for posting it when you did. I'm not trying to be a jerk or anything with these suggestions, just in the pursuit of making awesome things more awesome when I can.

 RSBuddyExchangeOracle.java

RSBuddyExchangePrice.java

I used Map because I wanted to construct something in parts, which I find to be more readable. Your change is more efficient, but now you're left with a constructor that has more than 3 parameters. I was persuaded in a coding video to value readability over efficiency, because I've all too often made the mistake of trying to hit the highest performance marks, yet the resulting code grows unwieldy and unmaintainable. Also, this oracle is dependent on RSBuddy not messing with their summary.json, which means the code has to be structured in a way that makes it easy to change parts that may break in the future. And we're only talking 1 second, which I don't think is a problem unless you're writing a GE investor script.

I used long values because an int overflow is to be expected when dealing with price * quantity, for example, if you were making a bank calculator, your bank value could exceed the highest int value and so the developer would have to cast from ints to longs retrospectively. I also did it because it coincides nicely with OSBot's own ItemContainer#getAmount too.

As for preferring to use names; be aware that some items share the same names. Although, none of the duplicate item names appear to be for any item with any significance, this could potentially be a problem in the future.

I'm glad you're changing it - it's why I put it out there. Make it your own. 

Edited January 3, 2019 by liverare