Indeed. It's the same effect you get from what are called "bang bang" position controllers that sense only whether the position is on one side of the taget or the other, and applies correcting actuator motion accordingly. The system tends to "hunt" around the set point rather than gliding smoothly to rest.
Sure, as I mentioned, one can include certain knowledge of the physical system in an intelligent controller to achieve better behavior.
What're the units of the analog of inductance in a thermal system? Electrical capacitance translates to thermal capacitance (Joules per K), resistance to thermal resistance (watts/meter/K). So what does thermal inductance look like? Can you name a material that exhibits it?
Guilty as charged.