JaGeX have control over the gaming environment, so I would assume they have systems set in motion that look out for large clusters of players (that aren't in consistently populated areas), and track what activity they're parking in (i.e., woodcutting) and how they interact with the gaming environment against the other fellow players.
I assume that they average out the interactions of everyone involved and see if there's a correlation between obvious botters. They probably map out movements on a 2D grid and see if there are oddities in paths that a large cluster of users/bots seem to exhibit routinely.
JaGeX probably graph out user interactions with in-game entities to see if there are common delays with particular players. While such data could be dismissed if it gathered from an isolated individual, I'd imagine that if the interaction patterns are frequent among multiple players, it's probably flagged as a call to concern.
I assume they verify their findings by sending in in-game mods to evaluate the environment and the users. Your script may check for mods, but that won't do shit. Remember; JaGeX control the game. It wouldn't be far-fetched to assume they have the ability to exist server-side only--if they wanted.
It's why I find anti-bans redundant. They don't do much. Jagex may be able to snipe out large clusters of bots in a single instance, but when it comes to bots who occupy the almost "outland" regions, I think they rely on other fellow players to file a report. ...Which is why I find anti-bans redundant. Once you're reported by a fellow player, it doesn't matter how much you shake your camera or hover over your skills, it's too late to "un-file" the report set against you. And you only justify said report if you continue to exhibit robot-like responses.