I don't think they detect the client itself. Instead I think they use individual actions to determine if you're botting or not. A majority of accounts banned are fresh ones, which are the easiest to detect imo. If we look at it from their perspective(or what I would do personally), they most likely flag you for certain things. They released in a post the average online time for the top 5 players which all averaged at 13 hours. If youre freshly made account logs in, does no quest, and goes directly to botting a single skill for >3-4 hours its like red alarms going off for Jagex. Or if you bot any single skill > average amount of hours, you're flagged. All this^ is just my opinion though, for all we know the client is detectable. The reason I think most of this is because we have those people who say "I've been botting for weeks and havent been banned." Yes it could be a delayed ban, but why weeks vs days?
Here's an example:
Assuming this is a fresh account, any account made that goes directly to training a skill non-stop(especially a resource skill) would just SCREAM bot to me if I were jagex. And if it were my game, that'd be something i'd take into consideration. (Might not ban immediately but a certain flag would be placed onto the account). If we just think more from their perspective I think the banrates would be cut in half.