Well, you must be doing something wrong. What tools do you have? A hand-held electric drill? A drill press? A millinbg machine? You won't get really good holes with hand-held. The right RPM for a specific drill size and workpiece material helps a lot. What aluminum alloy? Some drill very nicely, some are gummy and a big hassle. Are you using coolant? Even a drop of "thread cutting oil" from the hardware store will be a big help. The right feedrate for advancing the drill also helps.
I have drilled literally tens of thousands of holes in aluminum, mostly using a Bridgeport mill (since the mill can serve essentially as a drill press, I don't have a drill press.) I almost always use either coolant or cutting oil. I have used the same set of Indian-made drill bits I got from Harbor Freight 20 years ago. I occasionally resharpen them freehand on a bench grinder.
What you are doing is NOT considered hard at all. If you want a REALLY clean and round hole, use a reamer. Then use a countersink or deburring tool to clean up any ridge at the surface.
Jon