learning from digital inputs + vision

here's a killer app idea for a robot..  I'm very suprised it hasn't
been
done before..

webcam pointed at the TV screen, software tracks movement
of sprites on screen.

Video game system connected to TV.. Start with the Atari 2600
for simplicity of machine vision code..

Computer connected to webcam, and also has access to the
current state of the joystick (5 1-bit inputs.  Up, Down, Left, Right,
Button)

Software watches screen, relates movement on screen to joystick
state..

Software learns goal of game.  Restart in 2-player mode, and play
the computer.  If you program it correctly, it should be impossible
to beat the computer.

You could also do it without joystick feedback.  Let the computer
randomly cycle the joystick bits, and relate movement on the screen
to the joystick state.  It would learn how joystick relates to the
screen,
but not the concept of the game.

The point:  machine learning, and a friend to play games with.

Anyone interested in trying to get this work with me?  I have
a webcam, A PC, a TV, and an atari 2600.

You could allow the user to lasso parts of the screen, IE score
and remaining lives.  Perhaps a small programming language
within the software:

If lasso1 = previouslasso1 - 1 then BAD     --for lives
    GoLearn
 endif

if lasso1 = previouslasso1 + 1 then GOOD
    GoLearn
 endif


If lasso2 = previouslasso2 - 1 then BAD     --for score
    GoLearn
 endif


if lasso2 = previouslasso2 + 1 then GOOD
    GoLearn
 endif

Learn:
   what just happened on screen?

    Color change of objects?
         which objects?  store colors, shapes of objects, Good/BAD and
add them to a rule table
    collision between objects?
         which objects?  store colors, shapes of objects, Good/BAD and
add them to a rule table
    timer reached a certain count?
         which count?  store count and  Good/BAD  rule in rule table
    life indicator reached a certain count?
         which count?  store count and  Good/BAD  rule in rule table

then in the main loop:

     check for Closeness to Good rule
           move joystick to achieve good
     check for closeness to Bad Rule
           move joystick to get away from bad

PacMan seems like a good place to start

Rich