I remembered the PARC Smalltalk demo, but couldn't remember if it was the first implementation of multiple windows in an OS or not.
Smalltalk post-dated the xerox Alto by quite a few years.
But there was a fair amount of earlier work with text-only windows that is probably earlier than any of the bitmap display "windows." Even IBM mainframes with their 3270s had a relatively adbvanced display environment that one might call windows if you are in a charitable mood... (of course, it's not clear that that qualifies as "in the os.")
billw