Hi all, I am investigating the use/creation of a macro recorder / scripting tool that should be able to act as a human to manage a set of computers for daily tasks.
So far, there are different sw tools able to record and playback both keyboard and mouse actions, so nothing new.
I need to use something that will execute mouse and possibly keyboard actions while doing some processing before, passing then such actions to the OS.
There are tools able to recognize bitmaps within the screen, so I could detect if and where a particular button is, get its centroid and use the X, Y coordinates to accurately use the mouse to click on it.
I would like to automate the part when the program records the macro, so I would like that when I need to click on a button, I would press a "This click is supposed to be done over a button, please do a image capture of it" button on the mouse itself.
The recording software, would pass the click action to the OS, after having done a screen capture. After the click "execution", another screen capture would identify the area that is different and record it, each time I need it.
After that, I could create/refine the session having all the button bitmaps already taken with the right size, so I could search each bitmap and correctly position the mouse over it on each step.
I need a mouse emulator that will talk with my program, and my program will pass the mouse events to the OS but the mouse itself should not be detected as such to the OS.
The mouse cannot be detected as a real mouse to the OS, or I won't be able to intercept the clicks before the application receive them, so I could not perform actions like screen capture.
Important: I need to use the computers through RDP sessions, so pure graphics handling, 99% of the time, not local windows.
Just to make an example, if I just needed: left and right click, I could use the middle button for triggering the action needed using a PopUpMenu with the appropriate "commands", like LeftClickOnAButton vs SimpleLeftClick, so the program could take appropriate action before passing the leftclick to the OS.
Before starting reinventing the wheel, I was wondering if such device already exists, every emulator I found acts like a "normal" mouse or allows a pc to boot without keyboard.
TIA,
Giuseppe
Appl-to-Be-Automated