Botwatch is the name of the software Jagex uses to detect accounts using bots ingame. This system received a major update in September 2012 along with the release of Botany Bay. According to Mark Gerhard, Botwatch is able to profile accounts and determine whether or not they are using botting software. Most other anti-botting measures have focused on protecting and obfuscating RuneScape's source code, but Botwatch doesn't focus on breaking the code of the botting software. Rather, it monitors an account's actions and determines whether or not it is human behaviour. Certain checks are also in place to determine whether or not the game is being played using an official Jagex client.
The Botwatch system has banned players directly, making it the first automated banning system in RuneScape. This being said, Jagex is quite confident in this system and guarantees that all legitimate players will be safe.