Implementation of Human-like Mouse Movement

[blm95]

Hi guys,
I put together an algorithm which attempts to simulate mouse movement (at a fairly basic level).
 
The PoCs:
(10 pixel variation)
https://gfycat.com/BraveAnimatedGroundbeetle
https://gfycat.com/ThornyAfraidBedlingtonterrier
https://gfycat.com/FelineNippyDeinonychus
 
To show the effect of pixel variation:
(50 pixel variation)
https://gfycat.com/VastInformalAnaconda
 
Foreword:
Although some might say "mouse movements don't matter", it helps me sleep better at night knowing that my bots have decent anti-bans (placebo or not).
 
Explanation:
Human mouse movement can loosely be classified as a bezier curve (of course, some people move their mouses in a different fashion, but a bezier-curve motion seems to hold true for most people). The curvature is dependent upon the distance from the target location, as well as the relative Y location of the target.  Using the method written, it is possible to input a desired x/y position (implicit conversion to screen coordinates will come with time), and define the pixel variation for that action. For example, if the bounding box of an entity is 20x20, one can input a pixel variation of, say, 22 to typically hit the desired region, but occasionally miss (since being 100% accurate over the course of many hours might look sketchy).
 
Some Unknowns:
As you can see in the gifs, I attempt to slow down the mouse as it approaches the target area.  It is somewhat random with regards to when it slows down, but I am unsure whether I ought to make it slow down at increasing intervals every time (as it stands), or only occasionally slow the mouse down (with increased chances of it slowing down towards the end position).
 
I will not release anything yet, as it is obviously very early into development, but I'd like to get some feedback as to what I should improve/implement.  Although I think there are many more sophisticated, elegant, and human-like approaches to this problem, I think this is at least a good start over the built-in osbot mouse functionality.
 
Thanks,
blm95
 
edit: thanks to @Explv for the help with API

Edited November 14, 2015 by blm95

[Paradox68]

While this is really cool, let me contribute a few thoughts of my own that might help you expand.

 

Instead of always doing an arc, have it randomly switch which movement it does each time it is called between an arc, a stop-go (where it moves close to target diagonally, and then to the target), a bee-line (straight to target), and make the speed of the mouse randomize throughout the interactions so that mouse speed can not be a detected pattern. This would be a truly human-like mouse pattern and would probably be amazing to be able to add such a thing to scripts as it would improve anti-ban regardless of amount of uses due to the large randomization factor.

[blm95]

While this is really cool, let me contribute a few thoughts of my own that might help you expand.

 

Instead of always doing an arc, have it randomly switch which movement it does each time it is called between an arc, a stop-go (where it moves close to target diagonally, and then to the target), a bee-line (straight to target), and make the speed of the mouse randomize throughout the interactions so that mouse speed can not be a detected pattern. This would be a truly human-like mouse pattern and would probably be amazing to be able to add such a thing to scripts as it would improve anti-ban regardless of amount of uses due to the large randomization factor.

Thanks for the feedback .  For the stop-go, do you mean this? http://i.imgur.com/zgqykAX.png

[FrostBug]

Cool that you made this :E

But OSBot mouse movement atm looks more human-like than this tbh

[blm95]

Cool that you made this :E

But OSBot mouse movement atm looks more human-like than this tbh

Will be working more on it to try to make it look more and more human .

 

Here's a gif of it in action (really poor quality, not sure why; sorry): https://gfycat.com/DirtyTediousAnkolewatusi

[Flamezzz]

It's a fun project but bezier curves have been used in public bots for over 5 years, so if jagex does anything with the data they gather I assume the first thing they would do is try to fit a bezier curve. If you want human-like mouse movements I suggest you start with collecting actual human mouse data and train a model from there.

[Impensus]

human mouse movement would probably take the most direct path but due to using mouses etc it actually wobbles slightly and thus wouldnt be a straight line

[FrostBug]

Will be working more on it to try to make it look more and more human .

 

Here's a gif of it in action (really poor quality, not sure why; sorry): https://gfycat.com/DirtyTediousAnkolewatusi

 

That's pretty cool

 

Have you thought of adding something like acceleration and deacceleration? Start semi-slow, then accelerate the speed, and before eaching the target, start deaccelerating.

Then ofc. there's the aspect of "overshooting" the target. Eg. moving the cursor too far, or deaccelerating too soon.

 

[blm95]

It's a fun project but bezier curves have been used in public bots for over 5 years, so if jagex does anything with the data they gather I assume the first thing they would do is try to fit a bezier curve. If you want human-like mouse movements I suggest you start with collecting actual human mouse data and train a model from there.

 

 

That's pretty cool

 

Have you thought of adding something like acceleration and deacceleration? Start semi-slow, then accelerate the speed, and before eaching the target, start deaccelerating.

Then ofc. there's the aspect of "overshooting" the target. Eg. moving the cursor too far, or deaccelerating too soon.

 

As complexity increases, so does the difficulty in detection of course.  Keeping that in mind, I will try to implement more logic and randomness for different scenarios.  However, humans are not really random, so I'm not exactly sure at what point complexity and randomness actually begins to hinder the goal being achieved.  

 

Ideally, I would ask for thousands of users to run a script and play manually for X amount of hours (the more hours, the better the results).  While playing, it would record the entities/widgets/objects that were attempting to be hovered, and analyze their heights/widths, as well as beginning distance from target, initial mouse speed, and change in mouse speed over the course of interaction.  Using this, users could then use their own parsed data sets (which would be input into an ANN) for individual use (probably not a very good idea unless the individual recorded many many hours of gameplay), or use the ANN generated from all individuals.  In the end, the ANN would have some degree of randomness built in (maybe using data sets of 10-25 users each load?), as well as the ability for users to decide how accurate they'd like their bots to be (precision vs accuracy).

 

With that being said, a lot of work would have to go into that, and I'm pretty busy with uni lol.  Also, implementation of this would exceed my current level of understanding (although I think I would be able to learn pretty quickly).

 

Also, thanks to @Bobrocket for helping to discuss and brainstorm ideas for this project.

[nyan]

As complexity increases, so does the difficulty in detection of course.  Keeping that in mind, I will try to implement more logic and randomness for different scenarios.  However, humans are not really random, so I'm not exactly sure at what point complexity and randomness actually begins to hinder the goal being achieved.  

 

Ideally, I would ask for thousands of users to run a script and play manually for X amount of hours (the more hours, the better the results).  While playing, it would record the entities/widgets/objects that were attempting to be hovered, and analyze their heights/widths, as well as beginning distance from target, initial mouse speed, and change in mouse speed over the course of interaction.  Using this, users could then use their own parsed data sets (which would be input into an ANN) for individual use (probably not a very good idea unless the individual recorded many many hours of gameplay), or use the ANN generated from all individuals.  In the end, the ANN would have some degree of randomness built in (maybe using data sets of 10-25 users each load?), as well as the ability for users to decide how accurate they'd like their bots to be (precision vs accuracy).

 

With that being said, a lot of work would have to go into that, and I'm pretty busy with uni lol.  Also, implementation of this would exceed my current level of understanding (although I think I would be able to learn pretty quickly).

 

Also, thanks to @Bobrocket for helping to discuss and brainstorm ideas for this project.

 

When I was scripting for another website, one of the devs did a human mouse project. It was basically a quick program he made up that required the user to click on circles varying in size on the screen as soon as they appeared. I imagine you could do something similar and yield the same or similar results.

Edited November 14, 2015 by Shiny

[blm95]

When I was scripting for another website, one of the devs did a human mouse project. It was basically a quick program he made up that required the user to click on circles varying in size on the screen as soon as they appeared. I imagine you could do something similar and yield the same or similar results.

Got it basically done I think.  Obviously the data points need to be set more logically (with many more added), but here's the basis: https://gfycat.com/HonestUnfoldedEarwig.  Data is then serialized into an xml which can be used to parse useful bits of data to input into an algorithm: http://i.imgur.com/bGLrsT7.png Edited November 14, 2015 by blm95

[Paradox68]

Thanks for the feedback .  For the stop-go, do you mean this? http://i.imgur.com/zgqykAX.png

Yes but make the stop point closer to the target. and make it sometimes click in error before quickly clicking correctly afterwards. It's undeniably human-like that sometimes you accidentally don't click what you meant to.

Edited November 18, 2015 by Paradox68