To answer your question, Jagex CAN tell if you are using an unofficial client, but they just don't know which one. This is why they don't insta-ban if you use a botting client. As an experiment, try this: Play legit on one account, and at the same time, bot on another account, and see who gets banned first (or at all). Use the same client!
Mirror-mode is stealthy as fuck and Jagex thinks you are using the official client, when you are really botting.
Of course, this does not mean you can run shitty scripts and get away with it, you will need to run good scripts that mimic human behaviour to a 'T'.
NO ONE really knows how Jagex's detection system works so it is unknown what really flags an account. If they did use clients to flag people, then mirror-mode would certainly be useful and skip the initial detection phase. A lot of people have reported reduced bans with mirror-mode, so maybe it is helpful.