You're tapping the ends. How long are the pieces? Short enough to fit between the base of your drill press and the end of a tap in a tapping head?
A *hand* drill? Yes -- you are pretty likely to break them that way. They are quite brittle, and it is difficult to avoid putting side stress on the tap/drill when working that way.
And if you are drilling and tapping the *ends* of these extrusions, I suspect that these are not through holes. The drill/tap combinations are made for use in through holes as far as I can tell. The drill tip has to clear the far side of the workpiece before the tapping part starts working.
So -- you want a drill (in a drill chuck) followed by a spiral flute tap to get the chips out -- or even better a rolling tap (thread forming tap) which generates no chips -- but requires a slightly larger hole. (Look it up and order the right size drill bits at the same time. It is likely to be a Metric size. Look up the proper size in _Machinery's Handbook_.
This would still be better done in a drill press if the workpiece is short enough. If your drill press table will tilt to the side, rotate it 90 degrees, and bolt a chuck onto it to hold (and position) the workpieces. Drill them all, then change out the drill chuck for the tapping head and go back through tapping under power. Be sure to get the proper lubricant for the roll taps. But they will form stronger threads in your aluminum than the standard thread cutting taps will.
Now if you consider the drill/taps expensive, you *won't* like the new price of a tapping head. But they can be found somewhat more affordably on eBay -- given patience and luck. I got the smaller of mine (TapMatic) from eBay -- and the larger from someone else at a local metalworking club meeting.
You have my opinions above. Now to see what others have suggested.
Good luck, DoN.