The info on my McDonnell Douglas speed & feed calculator (slide rule sort of thing) says for end mills under 1", multiply chip load value by cutter dia. So, for aluminum, they recommend .010" feed per tooth at 1", so for a .125" diameter cutter, that should be .00125 (I'd be conservative, especially with HSS, you can watch the cutter bend under load.) So, 1/2" should be real safe at .005" feed per tooth.
I use something like 1/4 to one half of the feed rate you'd use for side cutting. When this is a straight plunge into solid material, there's no side force, but the chips tend to pile up. It may be better to make short stabs at it and then come up to clear the chips and apply coolant. Especially with aluminum, heating of the workpiece softens the aluminum, and then it sticks to the cutter, and you have a mess. So, you may have to allow cooling between cuts, use plenty of coolant, or other strategy. With a little experience you learn the kind of things that lead to the heat buildup. In some cases racing along with several cuts much shallower than rigidity dictates helps to keep the heat buildup from causing those messes. Also, knowing the alloys is important. 2024-T6 and 6061-T6 cuts like a dream, and you have to work at it to run into these troubles, like using a dull tool. But, 2024-T1 will drive you nuts, it is so soft it tears rather than cuts cleanly, and then the tool gets hot.
Yes, it really is a black art. The problem is no tool really cuts all the way to the center, as the center isn't turning. So, it has to mash the center out of the way and/or twist it to crumbs. The sharpness of the grinding work makes some difference, as it makes the area that doesn't cut smaller towards the infinitesimal. Take a look at some center-cutting end mills with a strong magnifier, and you will probably see differences, even from the same batch.
With larger endmills I push it until the machine starts to slow down or vibrate a little (not hard to do on a Clausing), then back off. I burned and dulled the first end mill while learning on a Bridgeport, then I knew when to lighten up. I've never broken an end mill but other prople have in company shops I ran. The corners chipped off, good reason to wear eye protection, but in these cases they weren't injured. I ground the end square past the break, added back relief almost to the edge, carefully reground the cutting edges (by hand & eye), and they were back milling again, though not nearly as fast.
The endmill downfeed rate is a lot higher if you drill a starting hole first. Or you can zig-zag down if the endmill isn't centercutting.