13 years ago
I am investigating the use/creation of a macro recorder / scripting tool
that should be able to act as a human to manage a set of computers for
So far, there are different sw tools able to record and playback both
keyboard and mouse actions, so nothing new.
I need to use something that will execute mouse and possibly keyboard
actions while doing some processing before, passing then such actions to
There are tools able to recognize bitmaps within the screen, so I could
detect if and where a particular button is, get its centroid and use the
X, Y coordinates to accurately use the mouse to click on it.
I would like to automate the part when the program records the macro, so
I would like that when I need to click on a button, I would press a
"This click is supposed to be done over a button, please do a image
capture of it" button on the mouse itself.
The recording software, would pass the click action to the OS, after
having done a screen capture.
After the click "execution", another screen capture would identify the
area that is different and record it, each time I need it.
After that, I could create/refine the session having all the button
bitmaps already taken with the right size, so I could search each bitmap
and correctly position the mouse over it on each step.
I need a mouse emulator that will talk with my program, and my program
will pass the mouse events to the OS but the mouse itself should not be
detected as such to the OS.
The mouse cannot be detected as a real mouse to the OS, or I won't be
able to intercept the clicks before the application receive them, so I
could not perform actions like screen capture.
Important: I need to use the computers through RDP sessions, so pure
graphics handling, 99% of the time, not local windows.
Just to make an example, if I just needed: left and right click, I could
use the middle button for triggering the action needed using a PopUpMenu
with the appropriate "commands", like LeftClickOnAButton vs
SimpleLeftClick, so the program could take appropriate action before
passing the leftclick to the OS.
Before starting reinventing the wheel, I was wondering if such device
already exists, every emulator I found acts like a "normal" mouse or
allows a pc to boot without keyboard.