OK, it's been a few years, but here's how I remember the next bit (please please please feel free to correct me):
When the metal is cold worked, these planes slide along each other. They slide until they either reach the end of the grain (where the planes don't match up) or until they hit an impurity large enough to stop them--like a carbon atom. Then it takes a slightly larger force to start the slide along the next available plane. You get enough work into the metal, and you run out of available room for the crystal planes to slide, and it gets brittle.
If you anneal the metal at this point, two things happen: You let all the tension flow out of the crystal planes, as it is much easier to get the atoms to flow over each other at this point and realign to however they need to be for minimum stored energy, and the grains tend to seperate into smaller grains along the larger stress disruptions in the crystal, up to a point. As you keep adding heat, the planes tend to realign to each other and you get bigger grains again.
When you work the metal hot, this doesn't matter nearlty so much, since it's so much easier for the atoms within the crystal to slide over each other that the strain never really builds up.
But for simple carbon steels, this is why some people like to do a little cold work, to deliberately build up some stress to toughen up the metal (use up some of the slide in the crystal planes) and why smaller grain structure can give better properties--it gives less room for the crsytals to "run" when subjected to stress.
I'm not sure how useful that is, but I always found it fascinating, at least! --Glenn Lyford