Hi Jeff:
Believe it or not, you can trace a tooth profile of almost any curve desired and then create a mating tooth that will drive with a smooth angular velocity. There is nothing special about an involute. As long as the common normal intersects the line of centers at a fixed point, the two tooth profiles are considered conjugate and the gears will mate with pure rolling contact. It just depends on how simple and interchangeable you want them to be.
Gears have been made in the past with several profile systems, the two standardized ones being the epicyclic and the involute. The involute, as you know, is generated by an unwinding "string" on the base circle; the epicyclic curve is generated by following the outer edge of a smaller circle rolling on the base circle. All modern industrial gearing is now of the involute type. Epicyclic gearing is used only in the manufacture of watch and clock gears. The biggest advantages of the involute system are ease of manufacture (an involute rack has straight faces) and the fact that a slight change in center distance will not affect the speed ratio between two gears.
OK, now lets talk about elliptical gears. The tooth profile of an involute gear could indeed be generated by wrapping a string around a "base ellipse", just like a circular one. And, in fact, this gear would mate with even an ordinary circular spur gear of similar pressure angle and pitch. Obviously since the base is an ellipse, the individual tooth profiles for each tooth will change depending on their location because the instantaneous radius of the base ellipse is not constant (as with an ordinary spur gear). For this reason it would be difficult to machine. It would probably require EDM machining, sintering, or extrusion.
I'm not sure what this rectangular gear you're talking about looks like, but I'm imagining that it is a rectangle with rounded corners. This would be an odd gear indeed. If the sides are straight they would be interlocking racks. Once they mesh you're screwed, since you lose angular driving power. A more practical (?) design would be with convex sides (like a TV screen). It would be the equivalent of several different circular spur gears. The convex sides would have their own pitch diameter and the corners would have another. As long as you keep the same diameteral pitch with all of them, you should have a perfectly good gear.
Don